+ PROC_LSST_SUBMIT_PATH=/mmfs1/home/stevengs/dirac/DEEP/submit + bps submit /gscratch/dirac/shared/opt/proc_lsst/pipelines/submit.yaml -b /mmfs1/home/stevengs/dirac/DEEP/repo -i DEEP/20211003/B1b --output-run DEEP/20211003/B1b/science#step0/20240318T051656Z --qgraph pipeline.qgraph lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process. lsst.ctrl.bps.drivers INFO: Starting submission process lsst.ctrl.bps.drivers INFO: Initializing execution environment lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 6.3639 seconds; current memory usage: 0.180 Gibyte, delta: 0.042 Gibyte, peak delta: 0.054 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.192 Gibyte (main), 0.000 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph) lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.0025 seconds lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.0064 seconds lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/pipeline_orig.qgraph DEEP/20211003/B1b/science#step0/20240318T051656Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/pipeline.qgraph lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 2.4986 seconds lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 6.6602 seconds; current memory usage: 0.273 Gibyte, delta: 0.092 Gibyte, peak delta: 0.080 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.273 Gibyte (main), 0.263 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs) lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0106 seconds; current memory usage: 0.273 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.273 Gibyte (main), 0.263 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 248 cluster(s) lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow) lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20211003_B1b_science#step0_20240318T051656Z' lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 0.0585 seconds; current memory usage: 0.273 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.273 Gibyte (main), 0.263 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 250 job(s) (including final) lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow) parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2 Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses s_addresses.add(address_by_interface(interface)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper retval = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface return socket.inet_ntoa(fcntl.ioctl( ^^^^^^^^^^^^ OSError: [Errno 99] Cannot assign requested address lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 0.2073 seconds; current memory usage: 0.274 Gibyte, delta: 0.001 Gibyte, peak delta: 0.001 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.274 Gibyte (main), 0.263 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting submit stage lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000 parsl.dataflow.dflow INFO: Starting DataFlowKernel with config Config( app_cache=True, checkpoint_files=None, checkpoint_mode='task_exit', checkpoint_period=None, executors=[MultiHighThroughputExecutor()], garbage_collect=True, initialize_logging=True, internal_tasks_max_threads=10, max_idletime=120.0, monitoring=None, retries=1, retry_handler=None, run_dir='runinfo', strategy='simple', usage_tracking=False ) parsl.dataflow.dflow INFO: Parsl version: 2023.06.12 parsl.usage_tracking.usage DEBUG: Tracking status: False parsl.dataflow.dflow INFO: Run id is: e5389a63-89c3-459a-8041-60f2b8715310 parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name parsl.dataflow.memoization INFO: App caching initialized parsl.dataflow.strategy DEBUG: Scaling strategy: simple parsl.executors.high_throughput.executor DEBUG: Starting queue management thread parsl.executors.high_throughput.executor DEBUG: queue management worker starting parsl.executors.high_throughput.executor DEBUG: Started queue management thread Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z parsl.executors.high_throughput.executor DEBUG: Created management thread: parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id={block_id} --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider: parsl.executors.status_handling INFO: Scaling out by 1 blocks parsl.executors.status_handling INFO: Allocated block ID 0 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0 2024-03-18 05:17:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 2024-03-18 05:17:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-18 05:17:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local parsl.providers.local.local DEBUG: Launching in remote mode 2024-03-18 05:17:31 proc_lsst.multi:170 [INFO] [multi] job_id 53289 proc_lsst.multi INFO: [multi] job_id 53289 2024-03-18 05:17:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:31 proc_lsst.multi:178 [INFO] [multi] provider local accepted submit and returned 53289 proc_lsst.multi INFO: [multi] provider local accepted submit and returned 53289 parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 53289 parsl.dataflow.job_status_poller DEBUG: Adding executor multi parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 0 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 0 has memoization hash 9292cc4565be5050e102e240b5372537 parsl.dataflow.memoization INFO: Task 0 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e79396980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1 parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/1fa494c2-b018-4153-a728-b8c3510f795d_isrForCrosstalkSources_1040013_31.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/1fa494c2-b018-4153-a728-b8c3510f795d_isrForCrosstalkSources_1040013_31.stderr parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1 has memoization hash c38a761148881ad983cba6300eda6f80 parsl.dataflow.memoization INFO: Task 1 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e79396ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 2 parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/9528ae46-79a5-49f1-b416-9382a8dfd657_isrForCrosstalkSources_1039907_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/9528ae46-79a5-49f1-b416-9382a8dfd657_isrForCrosstalkSources_1039907_24.stderr parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2 has memoization hash f880ab4e16b72592f6bdd8925deb2ead parsl.dataflow.memoization INFO: Task 2 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e79396fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 3 parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/2219ea56-9870-4069-b142-9844b9e7c4b7_isrForCrosstalkSources_1039907_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/2219ea56-9870-4069-b142-9844b9e7c4b7_isrForCrosstalkSources_1039907_61.stderr parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3 has memoization hash 0ccd79b7edd1db35380c92ce7852838f parsl.dataflow.memoization INFO: Task 3 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e79396e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 4 parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/4f1fb978-903d-4692-89a3-0765cbaa6167_isrForCrosstalkSources_1039909_18.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/4f1fb978-903d-4692-89a3-0765cbaa6167_isrForCrosstalkSources_1039909_18.stderr parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4 has memoization hash a473a5c2e95fe5209bb247a1ce6a0b69 parsl.dataflow.memoization INFO: Task 4 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e79396f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 5 parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/2faf5afa-8e6f-42e2-9ce3-b2c655f7ceef_isrForCrosstalkSources_1040013_39.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/2faf5afa-8e6f-42e2-9ce3-b2c655f7ceef_isrForCrosstalkSources_1040013_39.stderr parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5 has memoization hash b296462d428d40a3f30280757a6e306c parsl.dataflow.memoization INFO: Task 5 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e79397100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 6 parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/801f7551-0244-49fb-a550-c375ee167613_isrForCrosstalkSources_1039909_30.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/801f7551-0244-49fb-a550-c375ee167613_isrForCrosstalkSources_1039909_30.stderr parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 6 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 6 has memoization hash 024e87170a1c4c78c3952ceffaee86e0 parsl.dataflow.memoization INFO: Task 6 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e793971a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 7 parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/4909af3c-4287-45c4-afcc-5d880e43d3d8_isrForCrosstalkSources_1039908_18.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/4909af3c-4287-45c4-afcc-5d880e43d3d8_isrForCrosstalkSources_1039908_18.stderr parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 7 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 7 has memoization hash 69aeaa2aa598177a09ed054d243f33a3 parsl.dataflow.memoization INFO: Task 7 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e793962a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 8 parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/b189a951-59ae-4ff1-a739-bfae62bea25a_isrForCrosstalkSources_1039907_25.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/b189a951-59ae-4ff1-a739-bfae62bea25a_isrForCrosstalkSources_1039907_25.stderr parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 8 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 8 has memoization hash f569bc7214c87604ddcade1b4b54eb2b parsl.dataflow.memoization INFO: Task 8 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e79396b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 9 parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/e0f54475-421a-4f57-8943-5bb7b7096520_isrForCrosstalkSources_1039907_19.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/e0f54475-421a-4f57-8943-5bb7b7096520_isrForCrosstalkSources_1039907_19.stderr parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 9 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 9 has memoization hash 43211b29dfe3a352332abdab496d8f26 parsl.dataflow.memoization INFO: Task 9 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e793972e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 10 parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/08b32927-46a3-4ce8-8d95-4750a71160cf_isrForCrosstalkSources_1039908_55.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/08b32927-46a3-4ce8-8d95-4750a71160cf_isrForCrosstalkSources_1039908_55.stderr parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 10 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 10 has memoization hash 06fa38e294e42cd4135f8f53ec8b25f3 parsl.dataflow.memoization INFO: Task 10 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e79397060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 11 parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/437a8950-be02-479f-9641-2e2860915109_isrForCrosstalkSources_1039909_27.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/437a8950-be02-479f-9641-2e2860915109_isrForCrosstalkSources_1039909_27.stderr parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 11 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 11 has memoization hash b3331f5a2807ffd60f9c9c22d17318e2 parsl.dataflow.memoization INFO: Task 11 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e79397420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 12 parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/147d14b6-4d37-492d-b97a-ee8c6b68d0a2_isrForCrosstalkSources_1039907_20.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/147d14b6-4d37-492d-b97a-ee8c6b68d0a2_isrForCrosstalkSources_1039907_20.stderr parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 12 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 12 has memoization hash 6a23bdc851c561a0b4cedc7abe1886f6 parsl.dataflow.memoization INFO: Task 12 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e79397600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 13 parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/db465d62-ae44-425e-a290-8ef45f54d0ec_isrForCrosstalkSources_1040013_47.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/db465d62-ae44-425e-a290-8ef45f54d0ec_isrForCrosstalkSources_1040013_47.stderr parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 13 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 13 has memoization hash 4772b4b1f835b556be5995356eeef030 parsl.dataflow.memoization INFO: Task 13 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e793977e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 14 parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/fbdc779d-4994-44e8-b4ed-9b49040fcc9c_isrForCrosstalkSources_1039907_8.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/fbdc779d-4994-44e8-b4ed-9b49040fcc9c_isrForCrosstalkSources_1039907_8.stderr parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 14 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 14 has memoization hash 8879cba6421f639d0362f31329178330 parsl.dataflow.memoization INFO: Task 14 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e793979c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 15 parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/0e3920fc-fb96-4fcd-a8df-08dad46515b1_isrForCrosstalkSources_1039907_54.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/0e3920fc-fb96-4fcd-a8df-08dad46515b1_isrForCrosstalkSources_1039907_54.stderr parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 15 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 15 has memoization hash 43cb28c3303b66d736b045360ac3d1f6 parsl.dataflow.memoization INFO: Task 15 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e79397560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 16 parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/9de6dfe6-1773-4e88-b2e6-05040f8b01b5_isrForCrosstalkSources_1039909_26.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/9de6dfe6-1773-4e88-b2e6-05040f8b01b5_isrForCrosstalkSources_1039909_26.stderr parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 16 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 16 has memoization hash 6156504d21e6482f00236aac4b79e139 parsl.dataflow.memoization INFO: Task 16 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e79397a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 17 parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/fcca103c-d5fd-411e-bc1c-404542d6aeef_isrForCrosstalkSources_1039908_27.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/fcca103c-d5fd-411e-bc1c-404542d6aeef_isrForCrosstalkSources_1039908_27.stderr parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 17 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 17 has memoization hash 7fd1600935310c4c3631aa4683125599 parsl.dataflow.memoization INFO: Task 17 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e79397b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 18 parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/ad1761bd-6667-4577-a517-868c8ac9023d_isrForCrosstalkSources_1039908_19.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/ad1761bd-6667-4577-a517-868c8ac9023d_isrForCrosstalkSources_1039908_19.stderr parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 18 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 18 has memoization hash e69126f886d6a3ba30240d35514f05e0 parsl.dataflow.memoization INFO: Task 18 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e79397c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 19 parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/afaa291e-1057-442e-93b2-c9fc5f7b1e86_isrForCrosstalkSources_1039908_31.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/afaa291e-1057-442e-93b2-c9fc5f7b1e86_isrForCrosstalkSources_1039908_31.stderr parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 19 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 19 has memoization hash d4f3a6211284eaae1894516a5109da6a parsl.dataflow.memoization INFO: Task 19 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e79397920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 20 parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/a44e0614-264b-4444-98e2-825393ec13da_isrForCrosstalkSources_1039909_52.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/a44e0614-264b-4444-98e2-825393ec13da_isrForCrosstalkSources_1039909_52.stderr parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 20 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 20 has memoization hash a774e06f04192d4311eebc1b7319046c parsl.dataflow.memoization INFO: Task 20 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e79397d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 21 parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/facd5b4b-b4d8-4f12-896b-4c4e98d0a755_isrForCrosstalkSources_1040013_35.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/facd5b4b-b4d8-4f12-896b-4c4e98d0a755_isrForCrosstalkSources_1040013_35.stderr parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 21 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 21 has memoization hash 736f90b928f2ce68bc8dd5a25a78a57e parsl.dataflow.memoization INFO: Task 21 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e79397ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 22 parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/a92de5de-16ba-491c-a9a7-3db976b12b17_isrForCrosstalkSources_1039908_12.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/a92de5de-16ba-491c-a9a7-3db976b12b17_isrForCrosstalkSources_1039908_12.stderr parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 22 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash 4e48ee2d04a65363b05977b3e8040964 parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e79397e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 23 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/98679fe2-2616-4d7d-ba76-79919bca4c17_isrForCrosstalkSources_1040013_41.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/98679fe2-2616-4d7d-ba76-79919bca4c17_isrForCrosstalkSources_1040013_41.stderr parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 23 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 23 has memoization hash 2e48562e71396a9397d7d84984e3d525 parsl.dataflow.memoization INFO: Task 23 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e79397ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 23 try 0 launched on executor multi with executor id 24 parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/a1526ffd-bf44-44d5-afd1-3bf58c94b156_isrForCrosstalkSources_1039908_47.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/a1526ffd-bf44-44d5-afd1-3bf58c94b156_isrForCrosstalkSources_1039908_47.stderr parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 24 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 24 has memoization hash 600802e602cc5e6ccc7731cb7520d42c parsl.dataflow.memoization INFO: Task 24 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e79397ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 25 parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/f72fa67b-c3be-4709-9608-4d48ef0aada4_isrForCrosstalkSources_1039908_61.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/f72fa67b-c3be-4709-9608-4d48ef0aada4_isrForCrosstalkSources_1039908_61.stderr parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 25 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 25 has memoization hash 5e382376b46e3dc4f2867cc83b5a92cb parsl.dataflow.memoization INFO: Task 25 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e640e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 26 parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/63034363-3d3a-44ef-8a2f-f69b2f3f429c_isrForCrosstalkSources_1040013_30.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/63034363-3d3a-44ef-8a2f-f69b2f3f429c_isrForCrosstalkSources_1040013_30.stderr parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 26 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 26 has memoization hash ce571dcef8c0f79f74ff2fce2ab12645 parsl.dataflow.memoization INFO: Task 26 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e64220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 27 parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/606e4ae1-82cd-4a74-9b53-9deaa7e98678_isrForCrosstalkSources_1040013_5.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/606e4ae1-82cd-4a74-9b53-9deaa7e98678_isrForCrosstalkSources_1040013_5.stderr parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 27 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 27 has memoization hash d8f3507c265ae688fb2d83099833965a parsl.dataflow.memoization INFO: Task 27 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e64180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 28 parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/314cdf80-2ea2-41f7-8a46-75337853466c_isrForCrosstalkSources_1040013_15.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/314cdf80-2ea2-41f7-8a46-75337853466c_isrForCrosstalkSources_1040013_15.stderr parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 28 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash 6879595d3b67f4dcdadcdb3f6c4cd226 parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e642c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 29 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/ac7ef3a9-1e82-49cc-9dd4-39277772bfa6_isrForCrosstalkSources_1039907_41.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/ac7ef3a9-1e82-49cc-9dd4-39277772bfa6_isrForCrosstalkSources_1039907_41.stderr parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 29 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 29 has memoization hash 42e560c82fb993c4d5e72176a5c01bbb parsl.dataflow.memoization INFO: Task 29 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e64360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 29 try 0 launched on executor multi with executor id 30 parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/b2bf63ba-1d07-4a3c-9bdf-99525dc0f827_isrForCrosstalkSources_1039907_7.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/b2bf63ba-1d07-4a3c-9bdf-99525dc0f827_isrForCrosstalkSources_1039907_7.stderr parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 30 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 30 has memoization hash 5dc74189f1dfd48677cfd0fb0509733f parsl.dataflow.memoization INFO: Task 30 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e64400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 31 parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/aa03bb71-bc2e-41bc-8c9d-8d6d0dbcde2b_isrForCrosstalkSources_1039909_51.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/aa03bb71-bc2e-41bc-8c9d-8d6d0dbcde2b_isrForCrosstalkSources_1039909_51.stderr parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 31 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 31 has memoization hash 6073278a83eb0878c1debbe49b263aee parsl.dataflow.memoization INFO: Task 31 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e64540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 32 parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/f3cf710f-a847-4e0d-ab9e-311bafe29591_isrForCrosstalkSources_1039909_29.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/f3cf710f-a847-4e0d-ab9e-311bafe29591_isrForCrosstalkSources_1039909_29.stderr parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 32 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 32 has memoization hash 82d2cd3ab6ad0b28bc1b121ea3336998 parsl.dataflow.memoization INFO: Task 32 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e644a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 33 parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/885f8a7e-4636-46a4-879d-697bde2544f8_isrForCrosstalkSources_1039908_44.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/885f8a7e-4636-46a4-879d-697bde2544f8_isrForCrosstalkSources_1039908_44.stderr parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 33 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 33 has memoization hash 8dfad9e5fbbe54ad7eecb3e87a1f431d parsl.dataflow.memoization INFO: Task 33 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e645e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 34 parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/c28912f7-8aed-4d99-8f88-0416951a5142_isrForCrosstalkSources_1039907_31.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/c28912f7-8aed-4d99-8f88-0416951a5142_isrForCrosstalkSources_1039907_31.stderr parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 34 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash 68c1c0992b55eb0a6a84dceeb82a74a3 parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e64680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 35 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/804b6a4c-2c7d-41f6-a52a-ab30f493f66b_isrForCrosstalkSources_1039907_6.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/804b6a4c-2c7d-41f6-a52a-ab30f493f66b_isrForCrosstalkSources_1039907_6.stderr parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 35 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 35 has memoization hash af2103163dfe4346a62a96cb12d3e90c parsl.dataflow.memoization INFO: Task 35 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e64720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 35 try 0 launched on executor multi with executor id 36 parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/248f92e0-cb6c-4bd2-9ee8-9e8e690351a5_isrForCrosstalkSources_1040013_8.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/248f92e0-cb6c-4bd2-9ee8-9e8e690351a5_isrForCrosstalkSources_1040013_8.stderr parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 36 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 36 has memoization hash 55b4764c6edf56cb0ecec4133a7caa25 parsl.dataflow.memoization INFO: Task 36 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e64860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 37 parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/601b1e3e-a88e-476b-ba08-8234969cd0ce_isrForCrosstalkSources_1039907_48.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/601b1e3e-a88e-476b-ba08-8234969cd0ce_isrForCrosstalkSources_1039907_48.stderr parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 37 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 37 has memoization hash 013d9fcce33d6a0e2eda9d0ccd7951cf parsl.dataflow.memoization INFO: Task 37 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e647c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 38 parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/5228935c-738f-44e1-bd9f-106128f27123_isrForCrosstalkSources_1040013_33.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/5228935c-738f-44e1-bd9f-106128f27123_isrForCrosstalkSources_1040013_33.stderr parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 38 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 38 has memoization hash 79a695ba1e1cda8c33bccca65a537067 parsl.dataflow.memoization INFO: Task 38 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e64900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 39 parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/a0dc0534-f384-4501-8e5c-0b4ee3823213_isrForCrosstalkSources_1039908_43.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/a0dc0534-f384-4501-8e5c-0b4ee3823213_isrForCrosstalkSources_1039908_43.stderr parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 39 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 39 has memoization hash 8af6bfbc61a40d24a8ca7fe2b0f44939 parsl.dataflow.memoization INFO: Task 39 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e649a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 40 parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/0e3f3c7b-ad64-4068-b0c4-e6cdf50a7cff_isrForCrosstalkSources_1040013_1.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/0e3f3c7b-ad64-4068-b0c4-e6cdf50a7cff_isrForCrosstalkSources_1040013_1.stderr parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 40 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 40 has memoization hash d8cc9edd83f1818866bda214f23d08c7 parsl.dataflow.memoization INFO: Task 40 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e64a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 41 parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/45ced5c6-7552-4f53-943b-be8422b9bb9a_isrForCrosstalkSources_1039908_51.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/45ced5c6-7552-4f53-943b-be8422b9bb9a_isrForCrosstalkSources_1039908_51.stderr parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 41 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 41 has memoization hash 9d59065c3fd874dacb46a747e1af897c parsl.dataflow.memoization INFO: Task 41 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e64b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 42 parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/879ca815-0ad7-437c-9398-61b0b5203e44_isrForCrosstalkSources_1039907_62.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/879ca815-0ad7-437c-9398-61b0b5203e44_isrForCrosstalkSources_1039907_62.stderr parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 42 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 42 has memoization hash 0f1b8fe0516d431e3698b2d909eefae1 parsl.dataflow.memoization INFO: Task 42 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e64ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 43 parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/3640c35c-555c-4a84-a21d-06da2f35cc19_isrForCrosstalkSources_1039909_16.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/3640c35c-555c-4a84-a21d-06da2f35cc19_isrForCrosstalkSources_1039909_16.stderr parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 43 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 43 has memoization hash 8922ebf3803a40ff2bde8096d0341074 parsl.dataflow.memoization INFO: Task 43 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e64c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 44 parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/5ad1a495-3f86-454c-a8f4-2b9ac20635cf_isrForCrosstalkSources_1039908_16.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/5ad1a495-3f86-454c-a8f4-2b9ac20635cf_isrForCrosstalkSources_1039908_16.stderr parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 44 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 44 has memoization hash c23d06642cff937cc4aa5fa040988488 parsl.dataflow.memoization INFO: Task 44 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e64cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 45 parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/0b6d8f4c-1bab-4b57-908c-403f2bdd998c_isrForCrosstalkSources_1039907_35.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/0b6d8f4c-1bab-4b57-908c-403f2bdd998c_isrForCrosstalkSources_1039907_35.stderr parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 45 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 45 has memoization hash 3e6995742ed05cfc561c24e240ea10c0 parsl.dataflow.memoization INFO: Task 45 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e64d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 46 parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/a9e8ce2a-4c44-4e0f-ab69-7ad3e14423a7_isrForCrosstalkSources_1039907_51.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/a9e8ce2a-4c44-4e0f-ab69-7ad3e14423a7_isrForCrosstalkSources_1039907_51.stderr parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 46 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 46 has memoization hash d9b65582e9301580eb6d4038ca0ed4ce parsl.dataflow.memoization INFO: Task 46 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e64ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 47 parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/bdf18bee-c3ad-4cb5-992c-27bbabb33272_isrForCrosstalkSources_1039908_48.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/bdf18bee-c3ad-4cb5-992c-27bbabb33272_isrForCrosstalkSources_1039908_48.stderr parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 47 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 47 has memoization hash 93b39e929190f7832f4f02dc6268528b parsl.dataflow.memoization INFO: Task 47 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e64e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 48 parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/4781ae54-5d1c-4808-b481-f9caed6d35fc_isrForCrosstalkSources_1039909_7.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/4781ae54-5d1c-4808-b481-f9caed6d35fc_isrForCrosstalkSources_1039909_7.stderr parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 48 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 48 has memoization hash 275d3bd13279ea8c9f9eb3d8d404f0fe parsl.dataflow.memoization INFO: Task 48 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e64f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 49 parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/13ae3c66-ad45-4e99-924b-98b271e3bd82_isrForCrosstalkSources_1040013_44.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/13ae3c66-ad45-4e99-924b-98b271e3bd82_isrForCrosstalkSources_1040013_44.stderr parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 49 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 49 has memoization hash 61249d994f4727273d42860f6fb05e29 parsl.dataflow.memoization INFO: Task 49 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e64fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 50 parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/632d4347-3a68-4961-97c5-06d431910130_isrForCrosstalkSources_1039907_11.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/632d4347-3a68-4961-97c5-06d431910130_isrForCrosstalkSources_1039907_11.stderr parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 50 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 50 has memoization hash 74db28982ab2f1716d00e1b62357cfe2 parsl.dataflow.memoization INFO: Task 50 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e65080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 51 parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/6acb786b-61a6-4ff0-a145-d44960d8a8b7_isrForCrosstalkSources_1039909_14.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/6acb786b-61a6-4ff0-a145-d44960d8a8b7_isrForCrosstalkSources_1039909_14.stderr parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 51 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 51 has memoization hash 08969a243ec3896b0a2975133de6d9b8 parsl.dataflow.memoization INFO: Task 51 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e651c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 52 parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/4405fec8-2519-4f20-88bb-5425e09f3144_isrForCrosstalkSources_1039909_31.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/4405fec8-2519-4f20-88bb-5425e09f3144_isrForCrosstalkSources_1039909_31.stderr parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 52 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash 3f659561cd09941ff5a3fb762ecf3ea7 parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e65120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 53 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/fba9cb43-a684-4ef1-83c9-030768aef68e_isrForCrosstalkSources_1039907_42.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/fba9cb43-a684-4ef1-83c9-030768aef68e_isrForCrosstalkSources_1039907_42.stderr parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 53 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 53 has memoization hash 50c8e20f322f782401aa65c2fd34adf9 parsl.dataflow.memoization INFO: Task 53 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e65260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 53 try 0 launched on executor multi with executor id 54 parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/332509ab-57e7-4286-ad93-422efdd41e1c_isrForCrosstalkSources_1040013_2.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/332509ab-57e7-4286-ad93-422efdd41e1c_isrForCrosstalkSources_1040013_2.stderr parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 54 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 54 has memoization hash 8263710854b0a781293eb6c5817e171b parsl.dataflow.memoization INFO: Task 54 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e65300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 55 parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/6b3b8b2a-98da-4f75-8d66-aa4d35013c9a_isrForCrosstalkSources_1039908_21.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/6b3b8b2a-98da-4f75-8d66-aa4d35013c9a_isrForCrosstalkSources_1039908_21.stderr parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 55 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 55 has memoization hash 22d529b997d13c64211d2690a4b53b5e parsl.dataflow.memoization INFO: Task 55 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e653a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 56 parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/60138b96-53c3-4598-a106-51615c6fd984_isrForCrosstalkSources_1039908_28.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/60138b96-53c3-4598-a106-51615c6fd984_isrForCrosstalkSources_1039908_28.stderr parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 56 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 56 has memoization hash 7f3adc7781b3fb4031ac2d1932920d88 parsl.dataflow.memoization INFO: Task 56 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e654e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 57 parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/208e2b07-de2c-4d6d-8d77-db0a1a78cef1_isrForCrosstalkSources_1039909_32.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/208e2b07-de2c-4d6d-8d77-db0a1a78cef1_isrForCrosstalkSources_1039909_32.stderr parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 57 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 57 has memoization hash 07b183ba6cd63b799594c7c2fbaa3170 parsl.dataflow.memoization INFO: Task 57 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e65440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 58 parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/3cff5095-9540-489d-9d2e-65d0a295bf6c_isrForCrosstalkSources_1039909_55.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/3cff5095-9540-489d-9d2e-65d0a295bf6c_isrForCrosstalkSources_1039909_55.stderr parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 58 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 58 has memoization hash d6e252ee29b906267ab5fd420d1bef5d parsl.dataflow.memoization INFO: Task 58 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e65580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 59 parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/9185d49c-be8e-41b9-8dca-bbad79557879_isrForCrosstalkSources_1039908_1.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/9185d49c-be8e-41b9-8dca-bbad79557879_isrForCrosstalkSources_1039908_1.stderr parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 59 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 59 has memoization hash f6103710cf04990dd35c081090f6e395 parsl.dataflow.memoization INFO: Task 59 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e65620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 60 parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/b18cad74-fd87-4fba-a08a-8d063c662869_isrForCrosstalkSources_1039909_17.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/b18cad74-fd87-4fba-a08a-8d063c662869_isrForCrosstalkSources_1039909_17.stderr parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 60 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 60 has memoization hash 3e854d16b686f8d755bb8065b68e8611 parsl.dataflow.memoization INFO: Task 60 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e656c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 61 parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/388e8d06-56d7-4af3-8a38-c3c0cf53a2c5_isrForCrosstalkSources_1039908_50.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/388e8d06-56d7-4af3-8a38-c3c0cf53a2c5_isrForCrosstalkSources_1039908_50.stderr parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 61 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 61 has memoization hash 079f90f00cd4f92e9e6070d19ea181f4 parsl.dataflow.memoization INFO: Task 61 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e65800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 62 parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/fc7fa6ef-9610-414a-b44a-6195ac99afde_isrForCrosstalkSources_1039909_15.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/fc7fa6ef-9610-414a-b44a-6195ac99afde_isrForCrosstalkSources_1039909_15.stderr parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 62 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 62 has memoization hash d012688b2a8397caa8de1458bf38ae3a parsl.dataflow.memoization INFO: Task 62 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e65760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 63 parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/5044c9ba-588d-4a49-beee-2b9539748c13_isrForCrosstalkSources_1039909_56.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/5044c9ba-588d-4a49-beee-2b9539748c13_isrForCrosstalkSources_1039909_56.stderr parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 63 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 63 has memoization hash 2469ed45c39a2b5eb61b09e4d69a11dd parsl.dataflow.memoization INFO: Task 63 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e658a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 64 parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/d8983150-59e0-4526-8c24-b2d372d4110f_isrForCrosstalkSources_1039909_44.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/d8983150-59e0-4526-8c24-b2d372d4110f_isrForCrosstalkSources_1039909_44.stderr parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 64 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 64 has memoization hash 5bb91ed03c73be89202e8f7723a36660 parsl.dataflow.memoization INFO: Task 64 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e65940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 65 parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/83f60e4a-f9b4-4402-869a-ba7629753845_isrForCrosstalkSources_1039907_44.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/83f60e4a-f9b4-4402-869a-ba7629753845_isrForCrosstalkSources_1039907_44.stderr parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 65 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 65 has memoization hash ec8d2f1dbb911e54d2fdfc1af0db0f60 parsl.dataflow.memoization INFO: Task 65 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e659e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 65 try 0 launched on executor multi with executor id 66 parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/73d7d04f-71e4-4a86-8cd3-d098fab3742c_isrForCrosstalkSources_1040013_48.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/73d7d04f-71e4-4a86-8cd3-d098fab3742c_isrForCrosstalkSources_1040013_48.stderr parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 66 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 66 has memoization hash 0cbdb6974ac51cdea78e0d744fcc9898 parsl.dataflow.memoization INFO: Task 66 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e65b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 67 parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/2c89fb10-67ee-4007-a5e1-f54cc0cd7238_isrForCrosstalkSources_1039909_53.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/2c89fb10-67ee-4007-a5e1-f54cc0cd7238_isrForCrosstalkSources_1039909_53.stderr parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 67 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 67 has memoization hash d03c7d094ff363b436db84c11c3709cf parsl.dataflow.memoization INFO: Task 67 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e65a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 68 parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/c8af4291-1112-4ad3-a502-6765c790f5b2_isrForCrosstalkSources_1039908_24.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/c8af4291-1112-4ad3-a502-6765c790f5b2_isrForCrosstalkSources_1039908_24.stderr parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 68 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 68 has memoization hash fd4e0b29234d0a2a556858e00a737a2d parsl.dataflow.memoization INFO: Task 68 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e65bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 69 parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/f6471099-c717-4ade-b58e-eccfa20a634e_isrForCrosstalkSources_1039909_4.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/f6471099-c717-4ade-b58e-eccfa20a634e_isrForCrosstalkSources_1039909_4.stderr parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 69 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 69 has memoization hash d96c78a45ea0790f34cf7f25ef265fb9 parsl.dataflow.memoization INFO: Task 69 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e65c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 70 parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/b6349407-29c9-416b-88a5-9f0c1ccbfcfd_isrForCrosstalkSources_1039907_26.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/b6349407-29c9-416b-88a5-9f0c1ccbfcfd_isrForCrosstalkSources_1039907_26.stderr parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 70 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 70 has memoization hash 4fa4e651b7428e3b6d6201a1f46b067a parsl.dataflow.memoization INFO: Task 70 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e65d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 71 parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/0f03bec4-ef2b-48d4-8320-ba0d5c81b3e1_isrForCrosstalkSources_1039909_59.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/0f03bec4-ef2b-48d4-8320-ba0d5c81b3e1_isrForCrosstalkSources_1039909_59.stderr parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 71 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 71 has memoization hash 683861b84d0714c68847ef4b97d6a61c parsl.dataflow.memoization INFO: Task 71 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e65e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 71 try 0 launched on executor multi with executor id 72 parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/a0a75fb3-078e-4cc3-b00f-a713942db044_isrForCrosstalkSources_1039908_32.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/a0a75fb3-078e-4cc3-b00f-a713942db044_isrForCrosstalkSources_1039908_32.stderr parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 72 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 72 has memoization hash f929b2c7103cf61a25cd6c6e43416e87 parsl.dataflow.memoization INFO: Task 72 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e65da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 73 parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/96febe9e-3ea8-4c3f-87c3-18c952c03669_isrForCrosstalkSources_1040013_50.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/96febe9e-3ea8-4c3f-87c3-18c952c03669_isrForCrosstalkSources_1040013_50.stderr parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 73 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 73 has memoization hash b6f4b5a0da00e05bbd5e67faac568bac parsl.dataflow.memoization INFO: Task 73 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e65ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 74 parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/3f4f8971-68d6-4867-99a0-6c338ee76d3f_isrForCrosstalkSources_1039908_30.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/3f4f8971-68d6-4867-99a0-6c338ee76d3f_isrForCrosstalkSources_1039908_30.stderr parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 74 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 74 has memoization hash 6fd90375920ef91b3d18ad1f070fd8ba parsl.dataflow.memoization INFO: Task 74 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e65f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 75 parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/1d3b6cc3-db38-4b5c-8749-ace3a0c3a1a1_isrForCrosstalkSources_1039909_6.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/1d3b6cc3-db38-4b5c-8749-ace3a0c3a1a1_isrForCrosstalkSources_1039909_6.stderr parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 75 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 75 has memoization hash 3f84f8b7c8c3a552e78ca7edb3efdb3d parsl.dataflow.memoization INFO: Task 75 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e66020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 76 parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/72a58af0-9b9c-4b40-a920-1d674615286c_isrForCrosstalkSources_1039909_38.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/72a58af0-9b9c-4b40-a920-1d674615286c_isrForCrosstalkSources_1039909_38.stderr parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 76 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash b6cdcdf15d546b01bd606918a0c4b32d parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e66160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 77 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/7c12220d-b4ef-4047-8206-6a72afe05e33_isrForCrosstalkSources_1039909_60.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/7c12220d-b4ef-4047-8206-6a72afe05e33_isrForCrosstalkSources_1039909_60.stderr parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 77 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 77 has memoization hash 4aec33b8df9778b8f7d6ad6fe1e0ad55 parsl.dataflow.memoization INFO: Task 77 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e660c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 77 try 0 launched on executor multi with executor id 78 parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/41a170b3-f169-45ef-b591-ffbd73e736fe_isrForCrosstalkSources_1039909_58.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/41a170b3-f169-45ef-b591-ffbd73e736fe_isrForCrosstalkSources_1039909_58.stderr parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 78 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 78 has memoization hash a810fc65105d074c801e4baaa1a6c731 parsl.dataflow.memoization INFO: Task 78 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e66200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 79 parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/3d5958e0-f2dc-45ab-9f07-6bc8c0ccab1d_isrForCrosstalkSources_1039908_56.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/3d5958e0-f2dc-45ab-9f07-6bc8c0ccab1d_isrForCrosstalkSources_1039908_56.stderr parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 79 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 79 has memoization hash 8fe11efa30dc98c88575ed7517e78af8 parsl.dataflow.memoization INFO: Task 79 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e662a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 80 parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/5653a213-2721-43e5-9bfe-98a7dec7334f_isrForCrosstalkSources_1039909_5.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/5653a213-2721-43e5-9bfe-98a7dec7334f_isrForCrosstalkSources_1039909_5.stderr parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 80 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 80 has memoization hash cee26527dd377d11d2b4c12c8d98df91 parsl.dataflow.memoization INFO: Task 80 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e66340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 81 parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/2f0cfba8-9e82-4f6f-9b70-3682224eed0c_isrForCrosstalkSources_1040013_18.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/2f0cfba8-9e82-4f6f-9b70-3682224eed0c_isrForCrosstalkSources_1040013_18.stderr parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 81 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 81 has memoization hash 1cbae554b0afe1a4cbd9b53751e12409 parsl.dataflow.memoization INFO: Task 81 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e66480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 82 parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/e19c88ff-8f01-40c7-a181-996a7ffd3e20_isrForCrosstalkSources_1040013_60.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/e19c88ff-8f01-40c7-a181-996a7ffd3e20_isrForCrosstalkSources_1040013_60.stderr parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 82 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 82 has memoization hash b9d6e20e9223a9ca6b768c9e2c71f6dd parsl.dataflow.memoization INFO: Task 82 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e663e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 83 parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/f9ac4c6d-d633-4044-b6d1-1570d82ee032_isrForCrosstalkSources_1039909_22.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/f9ac4c6d-d633-4044-b6d1-1570d82ee032_isrForCrosstalkSources_1039909_22.stderr parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 83 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 83 has memoization hash c1fed8401e4e557e97f6e7499c51adc3 parsl.dataflow.memoization INFO: Task 83 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e66520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 83 try 0 launched on executor multi with executor id 84 parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/b7993104-e532-4702-b805-d319cfe03a54_isrForCrosstalkSources_1040013_11.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/b7993104-e532-4702-b805-d319cfe03a54_isrForCrosstalkSources_1040013_11.stderr parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 84 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 84 has memoization hash 696795cbd889adf752eed24753356c10 parsl.dataflow.memoization INFO: Task 84 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e665c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 85 parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/a69ba276-d84b-427a-a5d3-31279cde6a9c_isrForCrosstalkSources_1039908_26.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/a69ba276-d84b-427a-a5d3-31279cde6a9c_isrForCrosstalkSources_1039908_26.stderr parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 85 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 85 has memoization hash 26c506cb7a988830133d7ee7f3338dbc parsl.dataflow.memoization INFO: Task 85 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e66660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 86 parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/7b33a2af-5cbf-4980-888d-b2cbf59acc79_isrForCrosstalkSources_1039908_41.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/7b33a2af-5cbf-4980-888d-b2cbf59acc79_isrForCrosstalkSources_1039908_41.stderr parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 86 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 86 has memoization hash d9f1bc3dd9dbd72b26da929ea4de5199 parsl.dataflow.memoization INFO: Task 86 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e667a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 87 parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/a2334934-6daa-496e-90ab-4988cd430900_isrForCrosstalkSources_1040013_10.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/a2334934-6daa-496e-90ab-4988cd430900_isrForCrosstalkSources_1040013_10.stderr parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 87 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 87 has memoization hash 54fba683628aae716e26444e3d857180 parsl.dataflow.memoization INFO: Task 87 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e66700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 88 parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/a8335bab-9d41-4777-bead-83717ae172b7_isrForCrosstalkSources_1039909_41.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/a8335bab-9d41-4777-bead-83717ae172b7_isrForCrosstalkSources_1039909_41.stderr parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 88 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 88 has memoization hash 49ad55a65d169ce93b6af9832b5485d6 parsl.dataflow.memoization INFO: Task 88 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e66840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 89 parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/da2738f9-9bdc-40fa-a01c-42e7344aaa96_isrForCrosstalkSources_1039909_36.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/da2738f9-9bdc-40fa-a01c-42e7344aaa96_isrForCrosstalkSources_1039909_36.stderr parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 89 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 89 has memoization hash 50f4e72be47ba5bb0ae5073184004c77 parsl.dataflow.memoization INFO: Task 89 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e668e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 90 parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/e59f21aa-7a7f-4b80-a388-cda320736c65_isrForCrosstalkSources_1039909_42.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/e59f21aa-7a7f-4b80-a388-cda320736c65_isrForCrosstalkSources_1039909_42.stderr parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 90 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 90 has memoization hash 38de907fd73d5c7f35e25d60807a6165 parsl.dataflow.memoization INFO: Task 90 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e66980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 91 parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/00f7e4c1-9163-4e83-a426-9664540c7887_isrForCrosstalkSources_1039909_1.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/00f7e4c1-9163-4e83-a426-9664540c7887_isrForCrosstalkSources_1039909_1.stderr parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 91 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 91 has memoization hash 3311950ade074e9e28d1e9989cceeba2 parsl.dataflow.memoization INFO: Task 91 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e66ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 91 try 0 launched on executor multi with executor id 92 parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/7ba4f7b7-fc8c-4174-a142-ac551ccfc8e3_isrForCrosstalkSources_1040013_51.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/7ba4f7b7-fc8c-4174-a142-ac551ccfc8e3_isrForCrosstalkSources_1040013_51.stderr parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 92 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 92 has memoization hash 66672665f710b42ef38cdd0fa20a7ca4 parsl.dataflow.memoization INFO: Task 92 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e66a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 93 parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/f963eb80-c7b4-442a-911a-afb013859914_isrForCrosstalkSources_1039908_58.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/f963eb80-c7b4-442a-911a-afb013859914_isrForCrosstalkSources_1039908_58.stderr parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 93 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 93 has memoization hash e21c7da226ee51ee4ab83bc72a87a4bb parsl.dataflow.memoization INFO: Task 93 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e66b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 94 parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/1c162879-a26b-47b1-a3fe-d1cd8a899b14_isrForCrosstalkSources_1039909_12.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/1c162879-a26b-47b1-a3fe-d1cd8a899b14_isrForCrosstalkSources_1039909_12.stderr parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 94 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 94 has memoization hash 4bbce07c6b3d98990359cf9a36817243 parsl.dataflow.memoization INFO: Task 94 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e66c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 95 parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/6fdfc4e5-3b69-4980-be3c-dde03b68bc88_isrForCrosstalkSources_1039908_59.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/6fdfc4e5-3b69-4980-be3c-dde03b68bc88_isrForCrosstalkSources_1039908_59.stderr parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 95 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 95 has memoization hash 5040afd76698d1807894b39ec79c049b parsl.dataflow.memoization INFO: Task 95 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e66ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 96 parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/cbfd2566-3c7d-4669-a5b4-742e848fe527_isrForCrosstalkSources_1039908_9.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/cbfd2566-3c7d-4669-a5b4-742e848fe527_isrForCrosstalkSources_1039908_9.stderr parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 96 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 96 has memoization hash f66b9d5fbd008bef90b081f270b16b49 parsl.dataflow.memoization INFO: Task 96 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e66de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 97 parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/9caec26a-995a-4b03-8220-317666de406a_isrForCrosstalkSources_1039907_30.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/9caec26a-995a-4b03-8220-317666de406a_isrForCrosstalkSources_1039907_30.stderr parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 97 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 97 has memoization hash 0b58005a9a0e7da4fa7cdc156f01aecb parsl.dataflow.memoization INFO: Task 97 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e66d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 98 parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/38e3c569-113c-45f5-98af-bc7b7d179cf5_isrForCrosstalkSources_1040013_27.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/38e3c569-113c-45f5-98af-bc7b7d179cf5_isrForCrosstalkSources_1040013_27.stderr parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 98 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 98 has memoization hash 14ebeaa341123c6edac90e43cd26dd36 parsl.dataflow.memoization INFO: Task 98 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e66e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 99 parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/a7cb6420-09fb-407b-aba1-d7393e588ea8_isrForCrosstalkSources_1039908_5.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/a7cb6420-09fb-407b-aba1-d7393e588ea8_isrForCrosstalkSources_1039908_5.stderr parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 99 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 99 has memoization hash 3cdebb1f0640e7e667cfe6216df40436 parsl.dataflow.memoization INFO: Task 99 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e66f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 100 parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/bd4df630-f0fa-406a-aa70-92822fe3312f_isrForCrosstalkSources_1039909_21.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/bd4df630-f0fa-406a-aa70-92822fe3312f_isrForCrosstalkSources_1039909_21.stderr parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 100 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash 94cd655ec95182868248f0d262670def parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e66fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 101 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/eecfe7bc-f685-4a54-9b1a-d9c2cbf72bd4_isrForCrosstalkSources_1039909_40.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/eecfe7bc-f685-4a54-9b1a-d9c2cbf72bd4_isrForCrosstalkSources_1039909_40.stderr parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 101 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 101 has memoization hash fb8cd87783f3400dfb7aa4e0d16d8c34 parsl.dataflow.memoization INFO: Task 101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e67100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 101 try 0 launched on executor multi with executor id 102 parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/70eff367-ab19-443e-bc47-158016bbbaa6_isrForCrosstalkSources_1039908_33.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/70eff367-ab19-443e-bc47-158016bbbaa6_isrForCrosstalkSources_1039908_33.stderr parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 102 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 102 has memoization hash a21b334a7e94069bf0e14a3ce1324b12 parsl.dataflow.memoization INFO: Task 102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e67060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 103 parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/eef751d0-892b-4ac0-938f-ffa507de4079_isrForCrosstalkSources_1039907_21.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/eef751d0-892b-4ac0-938f-ffa507de4079_isrForCrosstalkSources_1039907_21.stderr parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 103 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 103 has memoization hash 3ba3bc52b967523d1c0ddfb99075923a parsl.dataflow.memoization INFO: Task 103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e671a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 104 parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/a83c5bc1-0c2f-4b96-9490-7493a94559dc_isrForCrosstalkSources_1039908_29.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/a83c5bc1-0c2f-4b96-9490-7493a94559dc_isrForCrosstalkSources_1039908_29.stderr parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 104 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 104 has memoization hash 9e0b28eb3089dcf924b0b91f99e8163b parsl.dataflow.memoization INFO: Task 104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e67240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 105 parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/7e0236ca-4d08-4c85-8ca5-77712ab31070_isrForCrosstalkSources_1039909_3.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/7e0236ca-4d08-4c85-8ca5-77712ab31070_isrForCrosstalkSources_1039909_3.stderr parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 105 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 105 has memoization hash 3c94034e60b1950a4aa051fc4106caf5 parsl.dataflow.memoization INFO: Task 105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e672e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 106 parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/86f7117f-1ce8-458f-8515-e1b681ab6a24_isrForCrosstalkSources_1039909_20.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/86f7117f-1ce8-458f-8515-e1b681ab6a24_isrForCrosstalkSources_1039909_20.stderr parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 106 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 106 has memoization hash d667ca6d8bcb31fb4b4c6098e5873b9c parsl.dataflow.memoization INFO: Task 106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e67420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 107 parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/5c965ba2-638c-44a8-be95-a1c250974166_isrForCrosstalkSources_1039908_23.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/5c965ba2-638c-44a8-be95-a1c250974166_isrForCrosstalkSources_1039908_23.stderr parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 107 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 107 has memoization hash 98392f01487f6a3d7ef75f9bb2d7722a parsl.dataflow.memoization INFO: Task 107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e67380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 108 parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/5bd88acd-c0b6-4fc3-bb33-b7c22ef45090_isrForCrosstalkSources_1039909_11.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/5bd88acd-c0b6-4fc3-bb33-b7c22ef45090_isrForCrosstalkSources_1039909_11.stderr parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 108 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 108 has memoization hash 7af50bd2a93da5fb5336142b7ea86435 parsl.dataflow.memoization INFO: Task 108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e674c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 109 parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/1520199e-2ca3-4452-bf76-9a18a5b9c02b_isrForCrosstalkSources_1039907_29.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/1520199e-2ca3-4452-bf76-9a18a5b9c02b_isrForCrosstalkSources_1039907_29.stderr parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 109 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 109 has memoization hash 3c945f9372ca0de45c1e62befa6b64ae parsl.dataflow.memoization INFO: Task 109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e67560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 110 parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/5034af32-c369-49be-9a23-488aa3b8f0c0_isrForCrosstalkSources_1039908_14.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/5034af32-c369-49be-9a23-488aa3b8f0c0_isrForCrosstalkSources_1039908_14.stderr parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 110 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 110 has memoization hash 2fc70f1a4813496c3de157220e2b1e6f parsl.dataflow.memoization INFO: Task 110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e67600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 111 parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/495b4694-0dee-4ca4-971f-af0c51268393_isrForCrosstalkSources_1039908_52.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/495b4694-0dee-4ca4-971f-af0c51268393_isrForCrosstalkSources_1039908_52.stderr parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 111 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 111 has memoization hash d43066392bd7f7b0320a04d3f93cd7b6 parsl.dataflow.memoization INFO: Task 111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e67740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 112 parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/794069d1-239a-41f2-a8bd-53478d58ef18_isrForCrosstalkSources_1039908_3.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/794069d1-239a-41f2-a8bd-53478d58ef18_isrForCrosstalkSources_1039908_3.stderr parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 112 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 112 has memoization hash 3683d8f302fb761a292890eb4dba3f9c parsl.dataflow.memoization INFO: Task 112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e676a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 113 parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/e063c6f3-0134-4f50-b5a9-1ac5ab33ce0e_isrForCrosstalkSources_1039908_36.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/e063c6f3-0134-4f50-b5a9-1ac5ab33ce0e_isrForCrosstalkSources_1039908_36.stderr parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 113 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 113 has memoization hash a8368312362f210586bebcddead069a4 parsl.dataflow.memoization INFO: Task 113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e677e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 114 parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/abf832e0-7bbf-4574-ac82-92bcc3a34d01_isrForCrosstalkSources_1039908_35.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/abf832e0-7bbf-4574-ac82-92bcc3a34d01_isrForCrosstalkSources_1039908_35.stderr parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 114 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 114 has memoization hash 5d751aeaa75549feda4f55f90009a01a parsl.dataflow.memoization INFO: Task 114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e67880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 115 parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/efbb0037-c71d-46af-bdfa-61a2f4012b19_isrForCrosstalkSources_1039909_33.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/efbb0037-c71d-46af-bdfa-61a2f4012b19_isrForCrosstalkSources_1039909_33.stderr parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 115 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 115 has memoization hash 3ecc161530ccc5eafcaadc18f463917e parsl.dataflow.memoization INFO: Task 115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e67920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 116 parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/29eb304f-c51a-45fb-a29a-25d4fbb0e0ec_isrForCrosstalkSources_1039909_57.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/29eb304f-c51a-45fb-a29a-25d4fbb0e0ec_isrForCrosstalkSources_1039909_57.stderr parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 116 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 116 has memoization hash 07e18a85076c7ff5cadca3b591a6480a parsl.dataflow.memoization INFO: Task 116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e67a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 117 parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/c46f2c59-462d-4e2c-b042-0c62c73b473b_isrForCrosstalkSources_1039907_13.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/c46f2c59-462d-4e2c-b042-0c62c73b473b_isrForCrosstalkSources_1039907_13.stderr parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 117 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 117 has memoization hash cb3a9c95deda43c62128bbceb9725729 parsl.dataflow.memoization INFO: Task 117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e679c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 118 parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/5c1f3cfe-d786-4541-ba4e-98e1c6d2aab3_isrForCrosstalkSources_1039909_45.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/5c1f3cfe-d786-4541-ba4e-98e1c6d2aab3_isrForCrosstalkSources_1039909_45.stderr parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 118 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash 587358d22fe8cd82f5636db26db09879 parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e67b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 119 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/952954f7-025c-4bef-9cf6-1e222a944736_isrForCrosstalkSources_1039909_48.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/952954f7-025c-4bef-9cf6-1e222a944736_isrForCrosstalkSources_1039909_48.stderr parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 119 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 119 has memoization hash 5a47c57b022f49781e685f6b5020fb3a parsl.dataflow.memoization INFO: Task 119 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e67ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 119 try 0 launched on executor multi with executor id 120 parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/c63671b8-ea23-4324-8fb6-7d83072b0f81_isrForCrosstalkSources_1040013_36.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/c63671b8-ea23-4324-8fb6-7d83072b0f81_isrForCrosstalkSources_1040013_36.stderr parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 120 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 120 has memoization hash 892dfd9e57980bc7f4589aef3866c848 parsl.dataflow.memoization INFO: Task 120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e67c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 121 parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/545bb221-d937-47df-9f0c-022d868d69cc_isrForCrosstalkSources_1039909_13.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/545bb221-d937-47df-9f0c-022d868d69cc_isrForCrosstalkSources_1039909_13.stderr parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 121 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 121 has memoization hash 5bbcf0053c3d3cd105220dfcd03e34a3 parsl.dataflow.memoization INFO: Task 121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e67d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 122 parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/7b7dd27f-c630-487e-916d-8114f186eecf_isrForCrosstalkSources_1040013_9.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/7b7dd27f-c630-487e-916d-8114f186eecf_isrForCrosstalkSources_1040013_9.stderr parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 122 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 122 has memoization hash 12c61b270f41a1bb95e42b34d662002c parsl.dataflow.memoization INFO: Task 122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e67ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 123 parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/11704442-474e-4c0d-ab30-c94ee1688071_isrForCrosstalkSources_1039907_39.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/11704442-474e-4c0d-ab30-c94ee1688071_isrForCrosstalkSources_1039907_39.stderr parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 123 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 123 has memoization hash 9f1887b03e0d5746f078c6faee15d597 parsl.dataflow.memoization INFO: Task 123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e67e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 124 parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/b8f80921-ba6b-45c7-bf69-1d9c3caf2479_isrForCrosstalkSources_1039907_57.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/b8f80921-ba6b-45c7-bf69-1d9c3caf2479_isrForCrosstalkSources_1039907_57.stderr parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 124 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 124 has memoization hash aa86bfff37a3a7cb9c4cab02fa5c2bac parsl.dataflow.memoization INFO: Task 124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e67ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 125 parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/5077302a-aa8e-4f96-a90b-39da603c8823_isrForCrosstalkSources_1039909_9.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/5077302a-aa8e-4f96-a90b-39da603c8823_isrForCrosstalkSources_1039909_9.stderr parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 125 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 125 has memoization hash 44bfda861f6f6075b2dde57e896c8f28 parsl.dataflow.memoization INFO: Task 125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e67f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 125 try 0 launched on executor multi with executor id 126 parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/6bce05dc-214f-4aee-9a9a-75c1382f1361_isrForCrosstalkSources_1040013_56.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/6bce05dc-214f-4aee-9a9a-75c1382f1361_isrForCrosstalkSources_1040013_56.stderr parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 126 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 126 has memoization hash 165c92c41c95d1d28660ee6656d4ce80 parsl.dataflow.memoization INFO: Task 126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78e64040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 127 parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/d4d3d269-c959-498f-8543-67da86401440_isrForCrosstalkSources_1039909_37.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/d4d3d269-c959-498f-8543-67da86401440_isrForCrosstalkSources_1039909_37.stderr parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 127 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 127 has memoization hash f31d06ef2b43624e422acc4847c01665 parsl.dataflow.memoization INFO: Task 127 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edc040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 127 try 0 launched on executor multi with executor id 128 parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/90736563-56c0-4754-a3e7-18f0be219029_isrForCrosstalkSources_1039908_22.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/90736563-56c0-4754-a3e7-18f0be219029_isrForCrosstalkSources_1039908_22.stderr parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 128 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 128 has memoization hash 6de41fe2426af97774ebf2458292c711 parsl.dataflow.memoization INFO: Task 128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edc180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 129 parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/3ca05ebb-b58e-45bb-a877-ec36645b9904_isrForCrosstalkSources_1039909_2.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/3ca05ebb-b58e-45bb-a877-ec36645b9904_isrForCrosstalkSources_1039909_2.stderr parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 129 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 129 has memoization hash 9923f3f16736f3a4487bf99aee85bef4 parsl.dataflow.memoization INFO: Task 129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edc220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 129 try 0 launched on executor multi with executor id 130 parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/5e3d4f6b-2683-41b5-9603-bb2a57a90131_isrForCrosstalkSources_1039908_54.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/5e3d4f6b-2683-41b5-9603-bb2a57a90131_isrForCrosstalkSources_1039908_54.stderr parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 130 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 130 has memoization hash 86a93fa04afbd6f31f4d37ca9d76454f parsl.dataflow.memoization INFO: Task 130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edc2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-18 05:17:32 proc_lsst.multi:146 [INFO] found job 53289 in provider local proc_lsst.multi INFO: found job 53289 in provider local parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 131 parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/7e667c07-9f08-41a6-a055-c166cd33dda1_isrForCrosstalkSources_1040013_16.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/7e667c07-9f08-41a6-a055-c166cd33dda1_isrForCrosstalkSources_1040013_16.stderr parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 131 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 131 has memoization hash 2cde382b89b606bab2d507b0127e2056 parsl.dataflow.memoization INFO: Task 131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edc400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 131 try 0 launched on executor multi with executor id 132 parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/8581258d-3f2d-49c5-a839-5b39208018aa_isrForCrosstalkSources_1039908_10.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/8581258d-3f2d-49c5-a839-5b39208018aa_isrForCrosstalkSources_1039908_10.stderr parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 132 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 132 has memoization hash e71629fc47ebad28a8d43ffb5bee6059 parsl.dataflow.memoization INFO: Task 132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edf920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 133 parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/e6c20bab-6380-4068-b504-c251585cdd7a_isrForCrosstalkSources_1039908_40.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/e6c20bab-6380-4068-b504-c251585cdd7a_isrForCrosstalkSources_1039908_40.stderr parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 133 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 133 has memoization hash f8a0f1bf3db8076390c41014dbdc2e53 parsl.dataflow.memoization INFO: Task 133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edc5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 133 try 0 launched on executor multi with executor id 134 parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/68783319-a47f-4548-9037-e066ba21c89f_isrForCrosstalkSources_1040013_34.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/68783319-a47f-4548-9037-e066ba21c89f_isrForCrosstalkSources_1040013_34.stderr parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 134 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 134 has memoization hash 3459a06a74e8fad6c9190a3ac3e365fb parsl.dataflow.memoization INFO: Task 134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edc4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 135 parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/f345aa69-44e8-48b3-901c-37413ad3e125_isrForCrosstalkSources_1039908_38.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/f345aa69-44e8-48b3-901c-37413ad3e125_isrForCrosstalkSources_1039908_38.stderr parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 135 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 135 has memoization hash 1e34a14970260744e6293823057f449c parsl.dataflow.memoization INFO: Task 135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edf9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 135 try 0 launched on executor multi with executor id 136 parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/0d05105e-d8a8-40ed-a92b-044b78d66992_isrForCrosstalkSources_1040013_58.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/0d05105e-d8a8-40ed-a92b-044b78d66992_isrForCrosstalkSources_1040013_58.stderr parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 136 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 136 has memoization hash cf558d570b1becad3bfce274e178e960 parsl.dataflow.memoization INFO: Task 136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edc720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 137 parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/d9aeb014-cf0b-4aa5-8715-56feac4189a0_isrForCrosstalkSources_1039908_57.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/d9aeb014-cf0b-4aa5-8715-56feac4189a0_isrForCrosstalkSources_1039908_57.stderr parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 137 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 137 has memoization hash 6108fabeb50d64958eaa932e3f1c74f7 parsl.dataflow.memoization INFO: Task 137 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edc9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 137 try 0 launched on executor multi with executor id 138 parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/9e389aa4-734b-4c65-97bc-91ef7c28cbae_isrForCrosstalkSources_1039909_19.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/9e389aa4-734b-4c65-97bc-91ef7c28cbae_isrForCrosstalkSources_1039909_19.stderr parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 138 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 138 has memoization hash 6b68c2e1bd9f038365834eeb32458a3d parsl.dataflow.memoization INFO: Task 138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edc900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 139 parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/7c21b7f5-ff62-4f47-a15c-69061f4f26aa_isrForCrosstalkSources_1040013_4.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/7c21b7f5-ff62-4f47-a15c-69061f4f26aa_isrForCrosstalkSources_1040013_4.stderr parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 139 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 139 has memoization hash 9bc8c7805c25534c66c2e09c62fe9060 parsl.dataflow.memoization INFO: Task 139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 139 try 0 launched on executor multi with executor id 140 parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/f3dcc77f-e2c5-471e-aabc-4d0a9dec93d9_isrForCrosstalkSources_1039908_17.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/f3dcc77f-e2c5-471e-aabc-4d0a9dec93d9_isrForCrosstalkSources_1039908_17.stderr parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 140 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 140 has memoization hash e257e7bcf1fe0e45823875d27c66f6fb parsl.dataflow.memoization INFO: Task 140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edc860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 141 parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/47419a5a-9165-4826-b748-90568a7bc3c6_isrForCrosstalkSources_1040013_13.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/47419a5a-9165-4826-b748-90568a7bc3c6_isrForCrosstalkSources_1040013_13.stderr parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 141 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 141 has memoization hash 2f626599b75cea9803b35eafc9da5a8c parsl.dataflow.memoization INFO: Task 141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edcae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 141 try 0 launched on executor multi with executor id 142 parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/e7d1b2fd-a600-4b19-8296-1d8590843609_isrForCrosstalkSources_1039908_62.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/e7d1b2fd-a600-4b19-8296-1d8590843609_isrForCrosstalkSources_1039908_62.stderr parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 142 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 142 has memoization hash 2a136be14da042d7250e4bdfed96c31b parsl.dataflow.memoization INFO: Task 142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edcc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 143 parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/8fd8bed7-68f3-45f3-84fa-a801840f674f_isrForCrosstalkSources_1039908_42.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/8fd8bed7-68f3-45f3-84fa-a801840f674f_isrForCrosstalkSources_1039908_42.stderr parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 143 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 143 has memoization hash 1ab17ff80e942ccb7a9bd0332285c9cc parsl.dataflow.memoization INFO: Task 143 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edcb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 143 try 0 launched on executor multi with executor id 144 parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/83aeaacc-3b57-41ba-a283-6010f377c51a_isrForCrosstalkSources_1039907_55.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/83aeaacc-3b57-41ba-a283-6010f377c51a_isrForCrosstalkSources_1039907_55.stderr parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 144 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 144 has memoization hash fc2a56dfcac6fa03fc99abc4f87513e9 parsl.dataflow.memoization INFO: Task 144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edcd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 145 parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/2b721a52-2ab1-42f1-87e4-91c9fd3d8378_isrForCrosstalkSources_1039907_53.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/2b721a52-2ab1-42f1-87e4-91c9fd3d8378_isrForCrosstalkSources_1039907_53.stderr parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 145 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 145 has memoization hash e5bc7ef44df381f70c1ba59d4c3aebba parsl.dataflow.memoization INFO: Task 145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 145 try 0 launched on executor multi with executor id 146 parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/2a6d8a5b-029c-4a5f-a07f-a9db04eb800d_isrForCrosstalkSources_1040013_32.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/2a6d8a5b-029c-4a5f-a07f-a9db04eb800d_isrForCrosstalkSources_1040013_32.stderr parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 146 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 146 has memoization hash 2cfa5fc032a539f1066405d76d11774c parsl.dataflow.memoization INFO: Task 146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 146 try 0 launched on executor multi with executor id 147 parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/1042bcfb-087f-4c28-8d99-541ed33e97cf_isrForCrosstalkSources_1039907_37.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/1042bcfb-087f-4c28-8d99-541ed33e97cf_isrForCrosstalkSources_1039907_37.stderr parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 147 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 147 has memoization hash 12ea48ea902cfcf81b97d29bac503829 parsl.dataflow.memoization INFO: Task 147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edcf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 147 try 0 launched on executor multi with executor id 148 parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/69f20d65-7fcf-46bb-bc45-dab41860ad20_isrForCrosstalkSources_1040013_28.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/69f20d65-7fcf-46bb-bc45-dab41860ad20_isrForCrosstalkSources_1040013_28.stderr parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 148 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 148 has memoization hash da235890b7b23d54dbae43ad8ff337aa parsl.dataflow.memoization INFO: Task 148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edcea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 149 parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/7659ddf9-17b0-44df-b254-819082ac99ff_isrForCrosstalkSources_1040013_49.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/7659ddf9-17b0-44df-b254-819082ac99ff_isrForCrosstalkSources_1040013_49.stderr parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 149 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 149 has memoization hash 8dc1b87314d9ef6160ea60dc520e0fec parsl.dataflow.memoization INFO: Task 149 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edcfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 149 try 0 launched on executor multi with executor id 150 parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/c0221f90-40c9-4d11-abf0-44c7da746c5e_isrForCrosstalkSources_1040013_21.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/c0221f90-40c9-4d11-abf0-44c7da746c5e_isrForCrosstalkSources_1040013_21.stderr parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 150 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 150 has memoization hash deb123a1d36267423e841c8cfe28772d parsl.dataflow.memoization INFO: Task 150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edd120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 151 parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/4bd448c6-e5cc-4c86-a3b8-46bb8d71dcd8_isrForCrosstalkSources_1039907_22.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/4bd448c6-e5cc-4c86-a3b8-46bb8d71dcd8_isrForCrosstalkSources_1039907_22.stderr parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 151 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 151 has memoization hash fb5352c52c222adf74466316421de143 parsl.dataflow.memoization INFO: Task 151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edd1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 151 parsl.dataflow.strategy DEBUG: Executor multi has 151 active tasks, 1/0 running/pending blocks, and 0 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 37 more blocks parsl.executors.status_handling INFO: Scaling out by 37 blocks parsl.executors.status_handling INFO: Allocated block ID 1 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1 2024-03-18 05:17:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 2024-03-18 05:17:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-18 05:17:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 151 try 0 launched on executor multi with executor id 152 parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/174dae44-b40b-4ab7-9f84-a43f73c3db5f_isrForCrosstalkSources_1039907_23.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/174dae44-b40b-4ab7-9f84-a43f73c3db5f_isrForCrosstalkSources_1039907_23.stderr parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 152 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 152 has memoization hash cc6f7f159fe5221c25445cb5a600776b parsl.dataflow.memoization INFO: Task 152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edd080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 152 try 0 launched on executor multi with executor id 153 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/ef0bc175-a1e7-4491-b98e-4f63cec33a60_isrForCrosstalkSources_1039907_17.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/ef0bc175-a1e7-4491-b98e-4f63cec33a60_isrForCrosstalkSources_1039907_17.stderr parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 153 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 153 has memoization hash 328af9961b73ad2dc195dbc5fe2a689a parsl.dataflow.memoization INFO: Task 153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edfe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 153 try 0 launched on executor multi with executor id 154 parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/94a20535-999c-4ed1-9f56-2d1f0bbe2678_isrForCrosstalkSources_1039907_1.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/94a20535-999c-4ed1-9f56-2d1f0bbe2678_isrForCrosstalkSources_1039907_1.stderr parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 154 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 154 has memoization hash ae8d051c871909af1cd9d5b70f72c0d3 parsl.dataflow.memoization INFO: Task 154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 155 parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/09b2e0d5-842f-4f2e-a675-55a41317ffb2_isrForCrosstalkSources_1039908_34.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/09b2e0d5-842f-4f2e-a675-55a41317ffb2_isrForCrosstalkSources_1039908_34.stderr parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 155 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 155 has memoization hash 1a386807c6f2c0ed63fac0163ce835c0 parsl.dataflow.memoization INFO: Task 155 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edd580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 155 try 0 launched on executor multi with executor id 156 parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/4291a003-df36-47f0-a027-21084ba66d97_isrForCrosstalkSources_1040013_43.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/4291a003-df36-47f0-a027-21084ba66d97_isrForCrosstalkSources_1040013_43.stderr parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 156 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 156 has memoization hash 0dcacf9523e5b94eb01fa7da351736da parsl.dataflow.memoization INFO: Task 156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edd6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 157 parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/9b75c933-fc24-4852-a00a-696ace6fcc71_isrForCrosstalkSources_1039907_45.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/9b75c933-fc24-4852-a00a-696ace6fcc71_isrForCrosstalkSources_1039907_45.stderr parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 157 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 157 has memoization hash 2389b38a77ddf94fb04e90133e972f59 parsl.dataflow.memoization INFO: Task 157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edd620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 157 try 0 launched on executor multi with executor id 158 parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/ed4140b5-a7c1-44a8-9d68-cbbc21beb7ae_isrForCrosstalkSources_1039908_49.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/ed4140b5-a7c1-44a8-9d68-cbbc21beb7ae_isrForCrosstalkSources_1039908_49.stderr parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 158 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 158 has memoization hash b4b121205085ba27f826db26d030bfaa parsl.dataflow.memoization INFO: Task 158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edfce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 158 try 0 launched on executor multi with executor id 159 parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/a43ed675-cc24-435b-814b-1b4ce243a229_isrForCrosstalkSources_1039908_8.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/a43ed675-cc24-435b-814b-1b4ce243a229_isrForCrosstalkSources_1039908_8.stderr parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 159 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 159 has memoization hash 7c45fdef28c35935ce1f23a802025fb6 parsl.dataflow.memoization INFO: Task 159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edd760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 159 try 0 launched on executor multi with executor id 160 parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/e2b903b4-d77d-411a-9c67-0fcec3b41623_isrForCrosstalkSources_1039908_7.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/e2b903b4-d77d-411a-9c67-0fcec3b41623_isrForCrosstalkSources_1039908_7.stderr parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 160 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 160 has memoization hash dbdfe5f721aa24ee16399920175f4647 parsl.dataflow.memoization INFO: Task 160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edd3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 160 try 0 launched on executor multi with executor id 161 parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/a1341065-7714-43bf-b14e-ecd2b7b18b0c_isrForCrosstalkSources_1039908_20.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/a1341065-7714-43bf-b14e-ecd2b7b18b0c_isrForCrosstalkSources_1039908_20.stderr parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 161 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 161 has memoization hash fd6e95faa0f3ccfa6b166d97b77e8b1e parsl.dataflow.memoization INFO: Task 161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edd8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 161 try 0 launched on executor multi with executor id 162 parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/fb5d3102-2fe6-4693-98d3-fe1f9c77fe5b_isrForCrosstalkSources_1040013_23.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/fb5d3102-2fe6-4693-98d3-fe1f9c77fe5b_isrForCrosstalkSources_1040013_23.stderr parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 162 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 162 has memoization hash b83620ede6e3408c999180a039ec1b7e parsl.dataflow.memoization INFO: Task 162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edd9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 163 parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/c362dec3-79c0-4fa6-8c24-fed9b120f3d2_isrForCrosstalkSources_1039907_3.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/c362dec3-79c0-4fa6-8c24-fed9b120f3d2_isrForCrosstalkSources_1039907_3.stderr parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 163 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 163 has memoization hash b882d1e7815661e616e96e2c453507c9 parsl.dataflow.memoization INFO: Task 163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edd800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 163 try 0 launched on executor multi with executor id 164 parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/27213552-4dc5-40f2-85b3-92d4f47b2b9a_isrForCrosstalkSources_1039909_49.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/27213552-4dc5-40f2-85b3-92d4f47b2b9a_isrForCrosstalkSources_1039909_49.stderr parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 164 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 164 has memoization hash 7c5520a9fcbcb879830941626c04f2e4 parsl.dataflow.memoization INFO: Task 164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edda80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 165 parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/e5d854ba-afa2-413d-8d5e-759e7d8b9b13_isrForCrosstalkSources_1040013_52.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/e5d854ba-afa2-413d-8d5e-759e7d8b9b13_isrForCrosstalkSources_1040013_52.stderr parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 165 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 165 has memoization hash ea835f9b9523156fe2572cf404af9bb5 parsl.dataflow.memoization INFO: Task 165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78eddbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 165 try 0 launched on executor multi with executor id 166 parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/5aa323a7-091b-4c55-b866-bc22b4f84079_isrForCrosstalkSources_1040013_59.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/5aa323a7-091b-4c55-b866-bc22b4f84079_isrForCrosstalkSources_1040013_59.stderr parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 166 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 166 has memoization hash 8761af9af7edaa93d33b9b63be6e406a parsl.dataflow.memoization INFO: Task 166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edd940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 167 parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/e6fb6c48-99bb-4603-9cc2-41c83e8488e7_isrForCrosstalkSources_1039909_24.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/e6fb6c48-99bb-4603-9cc2-41c83e8488e7_isrForCrosstalkSources_1039909_24.stderr parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 167 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 167 has memoization hash ad17c1541bbaabf17514ddf59bd3a797 parsl.dataflow.memoization INFO: Task 167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78eddd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 167 try 0 launched on executor multi with executor id 168 parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/733a4c35-236d-4296-ac5c-77043d6cbe6e_isrForCrosstalkSources_1039909_25.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/733a4c35-236d-4296-ac5c-77043d6cbe6e_isrForCrosstalkSources_1039909_25.stderr parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 168 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 168 has memoization hash 253090ee0d0553194a58839cecd99582 parsl.dataflow.memoization INFO: Task 168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78eddb20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 169 parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/583940cd-e64f-4823-b0f2-c244e242ac10_isrForCrosstalkSources_1039907_10.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/583940cd-e64f-4823-b0f2-c244e242ac10_isrForCrosstalkSources_1039907_10.stderr parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 169 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 169 has memoization hash 415939547ed681c2dbbfe680f70c66ff parsl.dataflow.memoization INFO: Task 169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78eddc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 169 try 0 launched on executor multi with executor id 170 parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/c4318a49-fdac-4561-8b4a-b84ae202ee09_isrForCrosstalkSources_1039907_28.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/c4318a49-fdac-4561-8b4a-b84ae202ee09_isrForCrosstalkSources_1039907_28.stderr parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 170 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 170 has memoization hash 1a0d32c194c32b776c4771c0bd130b6f parsl.dataflow.memoization INFO: Task 170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78eddee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 170 try 0 launched on executor multi with executor id 171 parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/4964450c-709f-4862-8e6a-3be38b740621_isrForCrosstalkSources_1039907_33.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/4964450c-709f-4862-8e6a-3be38b740621_isrForCrosstalkSources_1039907_33.stderr parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 171 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 171 has memoization hash cee90d14433280cdb21c848555144b12 parsl.dataflow.memoization INFO: Task 171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78eddda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 171 try 0 launched on executor multi with executor id 172 parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/ebed470b-b24c-43d4-82c0-c325cd66646c_isrForCrosstalkSources_1039909_23.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/ebed470b-b24c-43d4-82c0-c325cd66646c_isrForCrosstalkSources_1039909_23.stderr parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 172 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 172 has memoization hash 01e34fffb0ae16b63066d1ada6afa523 parsl.dataflow.memoization INFO: Task 172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edde40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 173 parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/742ef30b-effe-479b-b6bf-67258ff4aa52_isrForCrosstalkSources_1039908_53.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/742ef30b-effe-479b-b6bf-67258ff4aa52_isrForCrosstalkSources_1039908_53.stderr parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 173 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 173 has memoization hash 0a9782452f3a050e7ba84dc5fb2e6725 parsl.dataflow.memoization INFO: Task 173 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78ede020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 173 try 0 launched on executor multi with executor id 174 parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/22c621cc-9377-45e3-9883-df9655ae3e2c_isrForCrosstalkSources_1039907_36.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/22c621cc-9377-45e3-9883-df9655ae3e2c_isrForCrosstalkSources_1039907_36.stderr parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 174 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 174 has memoization hash 8c8d0d4e52781f88c34c23f80bd417fe parsl.dataflow.memoization INFO: Task 174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78ede0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 175 parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/8ea56a28-4c53-4beb-acf4-60c6d31d73d4_isrForCrosstalkSources_1039907_9.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/8ea56a28-4c53-4beb-acf4-60c6d31d73d4_isrForCrosstalkSources_1039907_9.stderr parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 175 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 175 has memoization hash 6373749e962978c17b8d5273f3ab18ac parsl.dataflow.memoization INFO: Task 175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78eddf80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 175 try 0 launched on executor multi with executor id 176 parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/7073fae5-c922-4c02-bf2f-1bad04bd6f24_isrForCrosstalkSources_1039908_25.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/7073fae5-c922-4c02-bf2f-1bad04bd6f24_isrForCrosstalkSources_1039908_25.stderr parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 176 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 176 has memoization hash a13861b0c2ef30dad2b857208465d7a3 parsl.dataflow.memoization INFO: Task 176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78ede2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 177 parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/095a7e0c-30c0-4828-a904-8baa4136e611_isrForCrosstalkSources_1039907_46.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/095a7e0c-30c0-4828-a904-8baa4136e611_isrForCrosstalkSources_1039907_46.stderr parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 177 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 177 has memoization hash 62c17519d7f49280be2e939ef730baef parsl.dataflow.memoization INFO: Task 177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78ede160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 177 try 0 launched on executor multi with executor id 178 parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/88f5c07e-e994-42bd-ada7-707306a9754f_isrForCrosstalkSources_1039907_40.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/88f5c07e-e994-42bd-ada7-707306a9754f_isrForCrosstalkSources_1039907_40.stderr parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 178 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 178 has memoization hash 853fe202149e14164bf4f6e857656371 parsl.dataflow.memoization INFO: Task 178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78ede340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 179 parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/af8a5c59-2a11-4f69-86f7-b414e0ef37e7_isrForCrosstalkSources_1039909_34.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/af8a5c59-2a11-4f69-86f7-b414e0ef37e7_isrForCrosstalkSources_1039909_34.stderr parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 179 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 179 has memoization hash 6b204d3edebc892a9cfeacc97349747f parsl.dataflow.memoization INFO: Task 179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78ede200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 179 try 0 launched on executor multi with executor id 180 parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/515bfa2a-8390-41a5-83f1-07e48ec89820_isrForCrosstalkSources_1040013_22.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/515bfa2a-8390-41a5-83f1-07e48ec89820_isrForCrosstalkSources_1040013_22.stderr parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 180 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 180 has memoization hash 8bd777e78d0b6efc649752931feddf37 parsl.dataflow.memoization INFO: Task 180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78ede3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 181 parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/00023a66-18f4-45bf-8dc9-565f4fd78d65_isrForCrosstalkSources_1039909_8.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/00023a66-18f4-45bf-8dc9-565f4fd78d65_isrForCrosstalkSources_1039909_8.stderr parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 181 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 181 has memoization hash 7c405b9f05ab633fb7bb603868c31891 parsl.dataflow.memoization INFO: Task 181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78ede480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 181 try 0 launched on executor multi with executor id 182 parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/70331391-8a80-4e88-b655-fa6965764102_isrForCrosstalkSources_1039907_58.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/70331391-8a80-4e88-b655-fa6965764102_isrForCrosstalkSources_1039907_58.stderr parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 182 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 182 has memoization hash be83056909a3df64ef21abc14c5f4267 parsl.dataflow.memoization INFO: Task 182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78ede520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 183 parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/ced119e1-d8aa-4017-b1db-1d5203f5e475_isrForCrosstalkSources_1039907_2.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/ced119e1-d8aa-4017-b1db-1d5203f5e475_isrForCrosstalkSources_1039907_2.stderr parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 183 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 183 has memoization hash 745178a06f90788e35e9d7011602a79f parsl.dataflow.memoization INFO: Task 183 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78ede660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 183 try 0 launched on executor multi with executor id 184 parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/687b093a-c930-491d-85e2-d7ee29126260_isrForCrosstalkSources_1040013_55.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/687b093a-c930-491d-85e2-d7ee29126260_isrForCrosstalkSources_1040013_55.stderr parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 184 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 184 has memoization hash b60738f09fd6d0a1f616812fa4959482 parsl.dataflow.memoization INFO: Task 184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edf880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 185 parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/f5a47ea5-cd46-48d1-808a-8571c8bf269f_isrForCrosstalkSources_1039908_13.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/f5a47ea5-cd46-48d1-808a-8571c8bf269f_isrForCrosstalkSources_1039908_13.stderr parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 185 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 185 has memoization hash 60f12620e79352a7147a83362741da6c parsl.dataflow.memoization INFO: Task 185 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edfa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 185 try 0 launched on executor multi with executor id 186 parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/38564380-f2ab-4239-b33a-3f8fd5a8bd8f_isrForCrosstalkSources_1040013_7.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/38564380-f2ab-4239-b33a-3f8fd5a8bd8f_isrForCrosstalkSources_1040013_7.stderr parsl.dataflow.dflow DEBUG: Task 186 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 186 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 186 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 186 has memoization hash 58deb565d486c52a395ea627e04b9f42 parsl.dataflow.memoization INFO: Task 186 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edf7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 186 try 0 launched on executor multi with executor id 187 parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/c19cd28b-693c-4003-ace6-aa6f3dcfc31e_isrForCrosstalkSources_1039908_45.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/c19cd28b-693c-4003-ace6-aa6f3dcfc31e_isrForCrosstalkSources_1039908_45.stderr parsl.dataflow.dflow DEBUG: Task 187 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 187 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 187 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 187 has memoization hash fe0141d342df54d75239095a75aca082 parsl.dataflow.memoization INFO: Task 187 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edf740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 187 try 0 launched on executor multi with executor id 188 parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/cb27bb6e-4b71-43df-99d8-a9231632fa3f_isrForCrosstalkSources_1039908_60.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/cb27bb6e-4b71-43df-99d8-a9231632fa3f_isrForCrosstalkSources_1039908_60.stderr parsl.dataflow.dflow DEBUG: Task 188 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 188 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 188 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 188 has memoization hash 48a2f39ebe982b825dfbaf0060b6562d parsl.dataflow.memoization INFO: Task 188 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edf6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 188 try 0 launched on executor multi with executor id 189 parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/4495dfe8-b121-4e89-a5a0-3f2fce30f312_isrForCrosstalkSources_1039907_43.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/4495dfe8-b121-4e89-a5a0-3f2fce30f312_isrForCrosstalkSources_1039907_43.stderr parsl.dataflow.dflow DEBUG: Task 189 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 189 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 189 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 189 has memoization hash f0f171820ddee2199a3043cded9d33f5 parsl.dataflow.memoization INFO: Task 189 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edf560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 189 try 0 launched on executor multi with executor id 190 parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/b684ebd1-e415-497a-b756-f23a8cfce56e_isrForCrosstalkSources_1039909_35.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/b684ebd1-e415-497a-b756-f23a8cfce56e_isrForCrosstalkSources_1039909_35.stderr parsl.dataflow.dflow DEBUG: Task 190 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 190 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 190 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 190 has memoization hash 0667b6d7880fe72211efcee050476b72 parsl.dataflow.memoization INFO: Task 190 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edf600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 190 try 0 launched on executor multi with executor id 191 parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/adacc621-147f-4b64-8cec-51016987f022_isrForCrosstalkSources_1039907_56.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/adacc621-147f-4b64-8cec-51016987f022_isrForCrosstalkSources_1039907_56.stderr parsl.dataflow.dflow DEBUG: Task 191 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 191 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 191 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 191 has memoization hash 9965eae20fe2e61a3c2b70e119e7f678 parsl.dataflow.memoization INFO: Task 191 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edf420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 191 try 0 launched on executor multi with executor id 192 parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/c1b4f062-9c1c-4986-9e46-a10e456a310e_isrForCrosstalkSources_1039907_12.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/c1b4f062-9c1c-4986-9e46-a10e456a310e_isrForCrosstalkSources_1039907_12.stderr parsl.dataflow.dflow DEBUG: Task 192 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 192 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 192 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 192 has memoization hash 71e7c5d6d5782c04785a5138d96af84a parsl.dataflow.memoization INFO: Task 192 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edf4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 192 try 0 launched on executor multi with executor id 193 parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/f791af7a-f105-4b6e-bc4a-4470dff713c1_isrForCrosstalkSources_1040013_53.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/f791af7a-f105-4b6e-bc4a-4470dff713c1_isrForCrosstalkSources_1040013_53.stderr parsl.dataflow.dflow DEBUG: Task 193 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 193 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 193 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 193 has memoization hash e161956c7023941adde6284467d9d6bd parsl.dataflow.memoization INFO: Task 193 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edf380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 193 try 0 launched on executor multi with executor id 194 parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/f946b7b1-387b-4834-88c6-f90f5495cd93_isrForCrosstalkSources_1039907_50.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/f946b7b1-387b-4834-88c6-f90f5495cd93_isrForCrosstalkSources_1039907_50.stderr parsl.dataflow.dflow DEBUG: Task 194 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 194 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 194 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 194 has memoization hash c1e824518b608c2c2f4a561f13e3f3b0 parsl.dataflow.memoization INFO: Task 194 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edf240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 194 try 0 launched on executor multi with executor id 195 parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/93488fd1-409e-4420-a988-ad769c90b84c_isrForCrosstalkSources_1039907_34.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/93488fd1-409e-4420-a988-ad769c90b84c_isrForCrosstalkSources_1039907_34.stderr parsl.dataflow.dflow DEBUG: Task 195 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 195 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 195 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 195 has memoization hash c2bb8438be922a28488eb6e6f06d6cf4 parsl.dataflow.memoization INFO: Task 195 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edf1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 195 try 0 launched on executor multi with executor id 196 parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/a1781469-d62f-494d-8c84-83c3d9d2b4e5_isrForCrosstalkSources_1040013_38.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/a1781469-d62f-494d-8c84-83c3d9d2b4e5_isrForCrosstalkSources_1040013_38.stderr parsl.dataflow.dflow DEBUG: Task 196 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 196 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 196 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 196 has memoization hash d0116da946b6e8d8b371a6c5b004881d parsl.dataflow.memoization INFO: Task 196 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edf100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 196 try 0 launched on executor multi with executor id 197 parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/dafcac66-bf6e-40db-8841-272363b40a6b_isrForCrosstalkSources_1039908_46.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/dafcac66-bf6e-40db-8841-272363b40a6b_isrForCrosstalkSources_1039908_46.stderr parsl.dataflow.dflow DEBUG: Task 197 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 197 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 197 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 197 has memoization hash 4d67be791962b32f703b3a4a028fb1d3 parsl.dataflow.memoization INFO: Task 197 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edefc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 197 try 0 launched on executor multi with executor id 198 parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/c476bda1-8f7f-4f4d-a86d-6036bbd32ee0_isrForCrosstalkSources_1040013_17.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/c476bda1-8f7f-4f4d-a86d-6036bbd32ee0_isrForCrosstalkSources_1040013_17.stderr parsl.dataflow.dflow DEBUG: Task 198 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 198 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 198 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 198 has memoization hash 2cb40268c2249db38119a11617fd4042 parsl.dataflow.memoization INFO: Task 198 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edf2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 198 try 0 launched on executor multi with executor id 199 parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/1e7d4758-448f-4be3-81c7-294d90508c5c_isrForCrosstalkSources_1039909_54.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/1e7d4758-448f-4be3-81c7-294d90508c5c_isrForCrosstalkSources_1039909_54.stderr parsl.dataflow.dflow DEBUG: Task 199 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 199 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 199 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 199 has memoization hash 8066c6a5ed61eb9210f43cc684c33dd0 parsl.dataflow.memoization INFO: Task 199 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 199 try 0 launched on executor multi with executor id 200 parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/ceeb8b8f-ca3e-438f-a279-3da290caaad4_isrForCrosstalkSources_1039909_61.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/ceeb8b8f-ca3e-438f-a279-3da290caaad4_isrForCrosstalkSources_1039909_61.stderr parsl.dataflow.dflow DEBUG: Task 200 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 200 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 200 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 200 has memoization hash ee3f90f42fac74d8b77c0b6cdad07dda parsl.dataflow.memoization INFO: Task 200 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 200 try 0 launched on executor multi with executor id 201 parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/176d2ff4-4871-4e3c-a608-676621539399_isrForCrosstalkSources_1039907_49.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/176d2ff4-4871-4e3c-a608-676621539399_isrForCrosstalkSources_1039907_49.stderr parsl.dataflow.dflow DEBUG: Task 201 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 201 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 201 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 201 has memoization hash d9892f05b378ca7e9bb820995d07ba6e parsl.dataflow.memoization INFO: Task 201 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edf060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 201 try 0 launched on executor multi with executor id 202 parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/a8fba6e9-f7cf-46a7-b8ac-1ce9e7d4a422_isrForCrosstalkSources_1039907_5.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/a8fba6e9-f7cf-46a7-b8ac-1ce9e7d4a422_isrForCrosstalkSources_1039907_5.stderr parsl.dataflow.dflow DEBUG: Task 202 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 202 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 202 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 202 has memoization hash 27d3c66727188bbdee480393efff4b76 parsl.dataflow.memoization INFO: Task 202 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78eded40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 202 try 0 launched on executor multi with executor id 203 parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/6e90343d-1517-4f55-99bf-32c32c21fcc5_isrForCrosstalkSources_1040013_45.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/6e90343d-1517-4f55-99bf-32c32c21fcc5_isrForCrosstalkSources_1040013_45.stderr parsl.dataflow.dflow DEBUG: Task 203 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 203 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 203 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 203 has memoization hash 0195ead259609a1c71b45cbc26559ff0 parsl.dataflow.memoization INFO: Task 203 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-18 05:17:33 proc_lsst.multi:170 [INFO] [multi] job_id 17194953 proc_lsst.multi INFO: [multi] job_id 17194953 2024-03-18 05:17:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:33 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17194953 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17194953 parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 17194953 parsl.executors.status_handling INFO: Allocated block ID 2 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2 2024-03-18 05:17:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 2024-03-18 05:17:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 203 try 0 launched on executor multi with executor id 204 parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/cac493bb-b3c2-4079-955b-a49124118f90_isrForCrosstalkSources_1039907_15.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/cac493bb-b3c2-4079-955b-a49124118f90_isrForCrosstalkSources_1039907_15.stderr parsl.dataflow.dflow DEBUG: Task 204 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 204 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 204 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 204 has memoization hash 09f9b87ca2004a9b333a07dac11b6225 parsl.dataflow.memoization INFO: Task 204 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 204 try 0 launched on executor multi with executor id 205 parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/aaec970b-f8e3-491b-8b36-15b0cdcf3689_isrForCrosstalkSources_1040013_57.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/aaec970b-f8e3-491b-8b36-15b0cdcf3689_isrForCrosstalkSources_1040013_57.stderr parsl.dataflow.dflow DEBUG: Task 205 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 205 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 205 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 205 has memoization hash 9ce3a218038a1dcf08be088a653f2bb1 parsl.dataflow.memoization INFO: Task 205 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edfd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 205 try 0 launched on executor multi with executor id 206 parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/2358a57d-de85-4700-80fa-2a7244c9cd85_isrForCrosstalkSources_1040013_29.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/2358a57d-de85-4700-80fa-2a7244c9cd85_isrForCrosstalkSources_1040013_29.stderr parsl.dataflow.dflow DEBUG: Task 206 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 206 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 206 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 206 has memoization hash ae21a9faeab28a7679a0cdfec881014b parsl.dataflow.memoization INFO: Task 206 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edfec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 206 try 0 launched on executor multi with executor id 207 parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/06622e6d-5953-43e7-8b87-43aab46f7056_isrForCrosstalkSources_1039907_47.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/06622e6d-5953-43e7-8b87-43aab46f7056_isrForCrosstalkSources_1039907_47.stderr parsl.dataflow.dflow DEBUG: Task 207 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 207 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 207 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 207 has memoization hash 776315bffce999babb8eea2bb2591d13 parsl.dataflow.memoization INFO: Task 207 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edd260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 207 try 0 launched on executor multi with executor id 208 parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/9ae58a90-9414-42a6-96ba-5fdb0d10503a_isrForCrosstalkSources_1039907_38.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/9ae58a90-9414-42a6-96ba-5fdb0d10503a_isrForCrosstalkSources_1039907_38.stderr parsl.dataflow.dflow DEBUG: Task 208 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 208 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 208 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 208 has memoization hash b102b973500804b48ca54ee32fec0745 parsl.dataflow.memoization INFO: Task 208 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edd4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 208 try 0 launched on executor multi with executor id 209 parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/9f6e756e-6e17-461b-b841-132755ac3896_isrForCrosstalkSources_1039908_39.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/9f6e756e-6e17-461b-b841-132755ac3896_isrForCrosstalkSources_1039908_39.stderr parsl.dataflow.dflow DEBUG: Task 209 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 209 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 209 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 209 has memoization hash 4f33d399fa99b733891e2aaaf7357190 parsl.dataflow.memoization INFO: Task 209 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edd300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 209 try 0 launched on executor multi with executor id 210 parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/b25222f8-ea97-4c38-bf39-2815156929aa_isrForCrosstalkSources_1040013_3.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/b25222f8-ea97-4c38-bf39-2815156929aa_isrForCrosstalkSources_1040013_3.stderr parsl.dataflow.dflow DEBUG: Task 210 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 210 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 210 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 210 has memoization hash cb9d9dbaf90608be916b8c61a251485f parsl.dataflow.memoization INFO: Task 210 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edc7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 210 try 0 launched on executor multi with executor id 211 parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/78f457b3-fd49-4231-b080-d1798e5d41f2_isrForCrosstalkSources_1039907_4.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/78f457b3-fd49-4231-b080-d1798e5d41f2_isrForCrosstalkSources_1039907_4.stderr parsl.dataflow.dflow DEBUG: Task 211 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 211 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 211 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 211 has memoization hash d7dcd2f6fcab19b8789f6afc3f467b68 parsl.dataflow.memoization INFO: Task 211 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78ede5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 211 try 0 launched on executor multi with executor id 212 parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/61b5faab-fab0-4494-bab0-b56f2de870e9_isrForCrosstalkSources_1039908_11.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/61b5faab-fab0-4494-bab0-b56f2de870e9_isrForCrosstalkSources_1039908_11.stderr parsl.dataflow.dflow DEBUG: Task 212 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 212 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 212 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 212 has memoization hash f1887285f565840cf43e3c531653abd8 parsl.dataflow.memoization INFO: Task 212 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edc0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 212 try 0 launched on executor multi with executor id 213 parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/64cba130-7c45-4e4f-b2f0-a129460a7314_isrForCrosstalkSources_1040013_26.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/64cba130-7c45-4e4f-b2f0-a129460a7314_isrForCrosstalkSources_1040013_26.stderr parsl.dataflow.dflow DEBUG: Task 213 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 213 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 213 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 213 has memoization hash 7b5686a603437339ebc6fc86de075ef5 parsl.dataflow.memoization INFO: Task 213 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78ede840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 213 try 0 launched on executor multi with executor id 214 parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/308a970f-6a5f-4cf0-86ad-87f294faf620_isrForCrosstalkSources_1039909_47.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/308a970f-6a5f-4cf0-86ad-87f294faf620_isrForCrosstalkSources_1039909_47.stderr parsl.dataflow.dflow DEBUG: Task 214 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 214 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 214 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 214 has memoization hash d75812df5e4129bde564aa366c208b50 parsl.dataflow.memoization INFO: Task 214 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78ede7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 214 try 0 launched on executor multi with executor id 215 parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/797cafeb-27c0-4738-93f9-8939064169d0_isrForCrosstalkSources_1039909_46.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/797cafeb-27c0-4738-93f9-8939064169d0_isrForCrosstalkSources_1039909_46.stderr parsl.dataflow.dflow DEBUG: Task 215 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 215 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 215 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 215 has memoization hash eff627bab069dcce7630a48b96ac2d3a parsl.dataflow.memoization INFO: Task 215 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78ede8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 215 try 0 launched on executor multi with executor id 216 parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/be24a218-0adb-4b90-88a9-6862c0064700_isrForCrosstalkSources_1040013_20.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/be24a218-0adb-4b90-88a9-6862c0064700_isrForCrosstalkSources_1040013_20.stderr parsl.dataflow.dflow DEBUG: Task 216 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 216 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 216 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 216 has memoization hash 9b9c75b60f19de3cbe1f586883295137 parsl.dataflow.memoization INFO: Task 216 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78ede700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 216 try 0 launched on executor multi with executor id 217 parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/3c6052c0-783d-4766-b17d-f63d78ae7a6f_isrForCrosstalkSources_1039909_50.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/3c6052c0-783d-4766-b17d-f63d78ae7a6f_isrForCrosstalkSources_1039909_50.stderr parsl.dataflow.dflow DEBUG: Task 217 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 217 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 217 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 217 has memoization hash 6fc49f0469ac45c86345903f8fced470 parsl.dataflow.memoization INFO: Task 217 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 217 try 0 launched on executor multi with executor id 218 parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/23fd06f2-c423-4979-813e-08ef08ce23df_isrForCrosstalkSources_1039908_6.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/23fd06f2-c423-4979-813e-08ef08ce23df_isrForCrosstalkSources_1039908_6.stderr parsl.dataflow.dflow DEBUG: Task 218 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 218 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 218 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 218 has memoization hash 1763bfd1b8a410ad801570c69803e8a9 parsl.dataflow.memoization INFO: Task 218 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78ede980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 218 try 0 launched on executor multi with executor id 219 parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/2ee61303-9db4-45c8-ac07-9b588878a965_isrForCrosstalkSources_1039907_16.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/2ee61303-9db4-45c8-ac07-9b588878a965_isrForCrosstalkSources_1039907_16.stderr parsl.dataflow.dflow DEBUG: Task 219 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 219 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 219 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 219 has memoization hash ef4a507eb1bb1067cdf4078dceb53012 parsl.dataflow.memoization INFO: Task 219 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edeac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 219 try 0 launched on executor multi with executor id 220 parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/0fd4bfcc-295f-438d-83bf-11fd70617747_isrForCrosstalkSources_1039907_18.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/0fd4bfcc-295f-438d-83bf-11fd70617747_isrForCrosstalkSources_1039907_18.stderr parsl.dataflow.dflow DEBUG: Task 220 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 220 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 220 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 220 has memoization hash a48d7cf0a48b9ce4700c60d6d8af842f parsl.dataflow.memoization INFO: Task 220 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edeca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 220 try 0 launched on executor multi with executor id 221 parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/6e2fc366-bb71-4de3-993b-328d808b9080_isrForCrosstalkSources_1039909_43.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/6e2fc366-bb71-4de3-993b-328d808b9080_isrForCrosstalkSources_1039909_43.stderr parsl.dataflow.dflow DEBUG: Task 221 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 221 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 221 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 221 has memoization hash 27b7ded2dfa09288c572d0365a465306 parsl.dataflow.memoization INFO: Task 221 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edd440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 221 try 0 launched on executor multi with executor id 222 parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/b1e82eec-2296-4096-950f-166799d246db_isrForCrosstalkSources_1040013_12.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/b1e82eec-2296-4096-950f-166799d246db_isrForCrosstalkSources_1040013_12.stderr parsl.dataflow.dflow DEBUG: Task 222 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 222 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 222 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 222 has memoization hash ab2b442f11a0ef7ef37c07056da0cb9c parsl.dataflow.memoization INFO: Task 222 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e78edfba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 222 try 0 launched on executor multi with executor id 223 parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/8a2db6f0-d447-4c25-abf0-3c7c620a4e70_isrForCrosstalkSources_1040013_19.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/8a2db6f0-d447-4c25-abf0-3c7c620a4e70_isrForCrosstalkSources_1040013_19.stderr parsl.dataflow.dflow DEBUG: Task 223 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 223 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 223 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 223 has memoization hash 9e19872edc02d69661e4ea3ab80cc2d6 parsl.dataflow.memoization INFO: Task 223 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e73d3c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 223 try 0 launched on executor multi with executor id 224 parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/51ef72c5-9243-4cf2-8de8-ffe57de96a18_isrForCrosstalkSources_1039908_2.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/51ef72c5-9243-4cf2-8de8-ffe57de96a18_isrForCrosstalkSources_1039908_2.stderr parsl.dataflow.dflow DEBUG: Task 224 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 224 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 224 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 224 has memoization hash 93df9dd1480aff469913daa6f35c9993 parsl.dataflow.memoization INFO: Task 224 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e73d3c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 224 try 0 launched on executor multi with executor id 225 parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/02134890-4c11-475a-88d4-870a79252eed_isrForCrosstalkSources_1039907_27.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/02134890-4c11-475a-88d4-870a79252eed_isrForCrosstalkSources_1039907_27.stderr parsl.dataflow.dflow DEBUG: Task 225 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 225 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 225 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 225 has memoization hash 03e5a1b1585f13457e4b1f85b8bb80b4 parsl.dataflow.memoization INFO: Task 225 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e73d3c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 225 try 0 launched on executor multi with executor id 226 parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/38376fc6-c154-4584-8ca1-b646e7c75991_isrForCrosstalkSources_1040013_40.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/38376fc6-c154-4584-8ca1-b646e7c75991_isrForCrosstalkSources_1040013_40.stderr parsl.dataflow.dflow DEBUG: Task 226 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 226 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 226 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 226 has memoization hash ee22ef8d064cbb60b47d35763e79e410 parsl.dataflow.memoization INFO: Task 226 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e73d3c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 226 try 0 launched on executor multi with executor id 227 parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/2dc37206-1fae-477f-83cb-0ff6c74a73c0_isrForCrosstalkSources_1040013_54.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/2dc37206-1fae-477f-83cb-0ff6c74a73c0_isrForCrosstalkSources_1040013_54.stderr parsl.dataflow.dflow DEBUG: Task 227 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 227 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 227 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 227 has memoization hash e0f9c5bff8dd79c6232ba74d1d83ca54 parsl.dataflow.memoization INFO: Task 227 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e73d3c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 227 try 0 launched on executor multi with executor id 228 parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/9e0fb149-0b41-462f-b97d-83bc0dbbbc27_isrForCrosstalkSources_1040013_61.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/9e0fb149-0b41-462f-b97d-83bc0dbbbc27_isrForCrosstalkSources_1040013_61.stderr parsl.dataflow.dflow DEBUG: Task 228 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 228 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 228 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 228 has memoization hash 30e4b737a4b280c74fc3bc9204a4d18e parsl.dataflow.memoization INFO: Task 228 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e73d3c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 228 try 0 launched on executor multi with executor id 229 parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/9b3051e1-2a6d-421d-af22-9af74216a6b9_isrForCrosstalkSources_1040013_62.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/9b3051e1-2a6d-421d-af22-9af74216a6b9_isrForCrosstalkSources_1040013_62.stderr parsl.dataflow.dflow DEBUG: Task 229 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 229 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 229 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 229 has memoization hash 7302e357adbefec16e9d93ea4a1909d3 parsl.dataflow.memoization INFO: Task 229 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e73d3c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 229 try 0 launched on executor multi with executor id 230 parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/3d56e608-4fac-46ab-85b1-605791d34a68_isrForCrosstalkSources_1039907_59.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/3d56e608-4fac-46ab-85b1-605791d34a68_isrForCrosstalkSources_1039907_59.stderr parsl.dataflow.dflow DEBUG: Task 230 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 230 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 230 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 230 has memoization hash 75ec5da1d7443b599d601c67aa3e05af parsl.dataflow.memoization INFO: Task 230 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e73d3c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 230 try 0 launched on executor multi with executor id 231 parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/bf47e2ad-e979-42dc-b8d7-bb0ad6ec84bf_isrForCrosstalkSources_1039908_15.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/bf47e2ad-e979-42dc-b8d7-bb0ad6ec84bf_isrForCrosstalkSources_1039908_15.stderr parsl.dataflow.dflow DEBUG: Task 231 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 231 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 231 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 231 has memoization hash eff6b786e760366b33ad5e553e642050 parsl.dataflow.memoization INFO: Task 231 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e73d3c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 231 try 0 launched on executor multi with executor id 232 parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/06596086-e281-4a11-9734-d20ca3d44548_isrForCrosstalkSources_1039908_37.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/06596086-e281-4a11-9734-d20ca3d44548_isrForCrosstalkSources_1039908_37.stderr parsl.dataflow.dflow DEBUG: Task 232 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 232 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 232 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 232 has memoization hash cdfb80d097b5f959d8dac5bacdc7e6d5 parsl.dataflow.memoization INFO: Task 232 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e73d3c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 232 try 0 launched on executor multi with executor id 233 parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/a14263de-846b-4c95-b680-43886dbb4f05_isrForCrosstalkSources_1039909_62.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/a14263de-846b-4c95-b680-43886dbb4f05_isrForCrosstalkSources_1039909_62.stderr parsl.dataflow.dflow DEBUG: Task 233 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 233 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 233 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 233 has memoization hash 5def4f6dc05edd5e0f826748efeeceb5 parsl.dataflow.memoization INFO: Task 233 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e73d3c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 233 try 0 launched on executor multi with executor id 234 parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/0f3d783f-5aa3-4a0a-b691-679636502183_isrForCrosstalkSources_1039909_10.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/0f3d783f-5aa3-4a0a-b691-679636502183_isrForCrosstalkSources_1039909_10.stderr parsl.dataflow.dflow DEBUG: Task 234 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 234 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 234 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 234 has memoization hash c96dbd2b4eef7639afbfafb12386469e parsl.dataflow.memoization INFO: Task 234 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e73d3c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 234 try 0 launched on executor multi with executor id 235 parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/7cfdd951-f0ea-49ec-9ebb-3ac84e73745a_isrForCrosstalkSources_1039909_39.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/7cfdd951-f0ea-49ec-9ebb-3ac84e73745a_isrForCrosstalkSources_1039909_39.stderr parsl.dataflow.dflow DEBUG: Task 235 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 235 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 235 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 235 has memoization hash ff4fe5e91813067f26616dba8fd90823 parsl.dataflow.memoization INFO: Task 235 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e73d3c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 235 try 0 launched on executor multi with executor id 236 parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/fe83e70e-9f22-463a-8ed1-429d2d3e69cc_isrForCrosstalkSources_1039907_60.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/fe83e70e-9f22-463a-8ed1-429d2d3e69cc_isrForCrosstalkSources_1039907_60.stderr parsl.dataflow.dflow DEBUG: Task 236 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 236 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 236 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 236 has memoization hash 7ad36f2b1151192a294e2c5846ae7579 parsl.dataflow.memoization INFO: Task 236 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e73d3c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 236 try 0 launched on executor multi with executor id 237 parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/6da9929f-00c7-4c0b-8914-5c7ef1569d86_isrForCrosstalkSources_1040013_42.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/6da9929f-00c7-4c0b-8914-5c7ef1569d86_isrForCrosstalkSources_1040013_42.stderr parsl.dataflow.dflow DEBUG: Task 237 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 237 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 237 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 237 has memoization hash 5e90da52f1482e33233467c89d58278f parsl.dataflow.memoization INFO: Task 237 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e73d3c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 237 try 0 launched on executor multi with executor id 238 parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/c6fbf300-66d1-44b5-8124-e3d0ac7aca06_isrForCrosstalkSources_1039908_4.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/c6fbf300-66d1-44b5-8124-e3d0ac7aca06_isrForCrosstalkSources_1039908_4.stderr parsl.dataflow.dflow DEBUG: Task 238 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 238 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 238 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 238 has memoization hash 9f3f33fdfe200364ad41d8fc6f992adc parsl.dataflow.memoization INFO: Task 238 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e73d3ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 238 try 0 launched on executor multi with executor id 239 parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/3b108fd4-e2b7-4617-9a72-afc60bd3771a_isrForCrosstalkSources_1039907_32.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/3b108fd4-e2b7-4617-9a72-afc60bd3771a_isrForCrosstalkSources_1039907_32.stderr parsl.dataflow.dflow DEBUG: Task 239 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 239 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 239 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 239 has memoization hash 78e51f3dd60b97882d03e578b0b8c10d parsl.dataflow.memoization INFO: Task 239 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e73d3cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 239 try 0 launched on executor multi with executor id 240 parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/8e187786-bca1-4cf8-acc2-cc0ae2e7b5ff_isrForCrosstalkSources_1040013_25.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/8e187786-bca1-4cf8-acc2-cc0ae2e7b5ff_isrForCrosstalkSources_1040013_25.stderr parsl.dataflow.dflow DEBUG: Task 240 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 240 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 240 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 240 has memoization hash 8645ff26da51eed82b0659b38e2f92cc parsl.dataflow.memoization INFO: Task 240 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e73d3cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 240 try 0 launched on executor multi with executor id 241 parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/e9153574-e001-4abf-be18-78af376a5db3_isrForCrosstalkSources_1040013_24.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/e9153574-e001-4abf-be18-78af376a5db3_isrForCrosstalkSources_1040013_24.stderr parsl.dataflow.dflow DEBUG: Task 241 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 241 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 241 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 241 has memoization hash 432f6d2c31461c48c2b85e10f20e4efd parsl.dataflow.memoization INFO: Task 241 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e73d3ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 241 try 0 launched on executor multi with executor id 242 parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/4fa6b11e-3d36-4900-9e49-58490df99fd7_isrForCrosstalkSources_1040013_6.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/4fa6b11e-3d36-4900-9e49-58490df99fd7_isrForCrosstalkSources_1040013_6.stderr parsl.dataflow.dflow DEBUG: Task 242 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 242 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 242 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 242 has memoization hash 56686eb7803d03948520a28cf9937f83 parsl.dataflow.memoization INFO: Task 242 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e73d3cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 242 try 0 launched on executor multi with executor id 243 parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/a7acc8d4-1a89-4bc0-8c68-8b39e409bdc4_isrForCrosstalkSources_1040013_46.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/a7acc8d4-1a89-4bc0-8c68-8b39e409bdc4_isrForCrosstalkSources_1040013_46.stderr parsl.dataflow.dflow DEBUG: Task 243 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 243 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 243 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 243 has memoization hash 849ed5e67a97ffae2385aa78ab039c77 parsl.dataflow.memoization INFO: Task 243 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e73d3cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 243 try 0 launched on executor multi with executor id 244 parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/f7fef0bf-9378-4cff-a681-b46c870569b9_isrForCrosstalkSources_1039907_14.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/f7fef0bf-9378-4cff-a681-b46c870569b9_isrForCrosstalkSources_1039907_14.stderr parsl.dataflow.dflow DEBUG: Task 244 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 244 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 244 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 244 has memoization hash 743cdfb18970e0c18c46960ba2257d38 parsl.dataflow.memoization INFO: Task 244 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e73d3ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 244 try 0 launched on executor multi with executor id 245 parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/7169de63-70e8-4109-9491-81a30117058d_isrForCrosstalkSources_1040013_37.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/7169de63-70e8-4109-9491-81a30117058d_isrForCrosstalkSources_1040013_37.stderr parsl.dataflow.dflow DEBUG: Task 245 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 245 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 245 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 245 has memoization hash a5c77dc7d4be2bd4b9a5b5436a2fbc5f parsl.dataflow.memoization INFO: Task 245 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e73d3cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 245 try 0 launched on executor multi with executor id 246 parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/781bc0f6-e4d5-4378-bb55-203575008c14_isrForCrosstalkSources_1039907_52.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/781bc0f6-e4d5-4378-bb55-203575008c14_isrForCrosstalkSources_1039907_52.stderr parsl.dataflow.dflow DEBUG: Task 246 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 246 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 246 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 246 has memoization hash 87ba494a9faf2972ceda8179dff01531 parsl.dataflow.memoization INFO: Task 246 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e73d3cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 246 try 0 launched on executor multi with executor id 247 parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/69e2eae2-6728-414a-9058-ac450645e98f_isrForCrosstalkSources_1040013_14.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/69e2eae2-6728-414a-9058-ac450645e98f_isrForCrosstalkSources_1040013_14.stderr parsl.dataflow.dflow DEBUG: Task 247 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 247 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 247 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 247 has memoization hash 95c88ee61d51d0e8e1e08a541195e2fd parsl.dataflow.memoization INFO: Task 247 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x150e73d3cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 247 try 0 launched on executor multi with executor id 248 parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/0a8cccd3-faaa-4e8c-8412-1f9085fe8d99_isrForCrosstalkSources_1039909_28.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/0a8cccd3-faaa-4e8c-8412-1f9085fe8d99_isrForCrosstalkSources_1039909_28.stderr 2024-03-18 05:17:34 proc_lsst.multi:170 [INFO] [multi] job_id 17194954 proc_lsst.multi INFO: [multi] job_id 17194954 2024-03-18 05:17:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-18 05:17:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:34 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17194954 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17194954 parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 17194954 parsl.executors.status_handling INFO: Allocated block ID 3 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3 2024-03-18 05:17:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 2024-03-18 05:17:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-18 05:17:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:34 proc_lsst.multi:170 [INFO] [multi] job_id 17194955 proc_lsst.multi INFO: [multi] job_id 17194955 2024-03-18 05:17:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-18 05:17:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:34 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17194955 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17194955 parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 17194955 parsl.executors.status_handling INFO: Allocated block ID 4 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4 2024-03-18 05:17:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 2024-03-18 05:17:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-18 05:17:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:34 proc_lsst.multi:170 [INFO] [multi] job_id 17194956 proc_lsst.multi INFO: [multi] job_id 17194956 2024-03-18 05:17:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-18 05:17:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:34 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17194956 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17194956 parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 17194956 parsl.executors.status_handling INFO: Allocated block ID 5 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5 2024-03-18 05:17:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 2024-03-18 05:17:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-18 05:17:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:35 proc_lsst.multi:170 [INFO] [multi] job_id 17194957 proc_lsst.multi INFO: [multi] job_id 17194957 2024-03-18 05:17:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:17:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:35 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17194957 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17194957 parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 17194957 parsl.executors.status_handling INFO: Allocated block ID 6 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6 2024-03-18 05:17:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 2024-03-18 05:17:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:17:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:35 proc_lsst.multi:170 [INFO] [multi] job_id 17194958 proc_lsst.multi INFO: [multi] job_id 17194958 2024-03-18 05:17:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-18 05:17:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:35 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17194958 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17194958 parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 17194958 parsl.executors.status_handling INFO: Allocated block ID 7 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7 2024-03-18 05:17:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 2024-03-18 05:17:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-18 05:17:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:35 proc_lsst.multi:170 [INFO] [multi] job_id 17194959 proc_lsst.multi INFO: [multi] job_id 17194959 2024-03-18 05:17:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-18 05:17:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:35 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17194959 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17194959 parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 17194959 parsl.executors.status_handling INFO: Allocated block ID 8 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8 2024-03-18 05:17:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 2024-03-18 05:17:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-18 05:17:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:36 proc_lsst.multi:170 [INFO] [multi] job_id 17194960 proc_lsst.multi INFO: [multi] job_id 17194960 2024-03-18 05:17:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-18 05:17:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:36 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17194960 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17194960 parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 17194960 parsl.executors.status_handling INFO: Allocated block ID 9 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9 2024-03-18 05:17:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 2024-03-18 05:17:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-18 05:17:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:36 proc_lsst.multi:170 [INFO] [multi] job_id 17194961 proc_lsst.multi INFO: [multi] job_id 17194961 2024-03-18 05:17:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-18 05:17:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:36 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17194961 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17194961 parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 17194961 parsl.executors.status_handling INFO: Allocated block ID 10 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10 2024-03-18 05:17:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 2024-03-18 05:17:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-18 05:17:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:36 proc_lsst.multi:170 [INFO] [multi] job_id 17194962 proc_lsst.multi INFO: [multi] job_id 17194962 2024-03-18 05:17:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-18 05:17:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:36 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17194962 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17194962 parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 17194962 parsl.executors.status_handling INFO: Allocated block ID 11 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11 2024-03-18 05:17:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 2024-03-18 05:17:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-18 05:17:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:37 proc_lsst.multi:170 [INFO] [multi] job_id 17194963 proc_lsst.multi INFO: [multi] job_id 17194963 2024-03-18 05:17:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-18 05:17:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:37 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17194963 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17194963 parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 17194963 parsl.executors.status_handling INFO: Allocated block ID 12 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12 2024-03-18 05:17:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 2024-03-18 05:17:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-18 05:17:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:37 proc_lsst.multi:170 [INFO] [multi] job_id 17194964 proc_lsst.multi INFO: [multi] job_id 17194964 2024-03-18 05:17:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-18 05:17:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:37 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17194964 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17194964 parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 17194964 parsl.executors.status_handling INFO: Allocated block ID 13 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13 2024-03-18 05:17:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 2024-03-18 05:17:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-18 05:17:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:37 proc_lsst.multi:170 [INFO] [multi] job_id 17194965 proc_lsst.multi INFO: [multi] job_id 17194965 2024-03-18 05:17:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-18 05:17:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:37 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17194965 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17194965 parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 17194965 parsl.executors.status_handling INFO: Allocated block ID 14 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14 2024-03-18 05:17:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 2024-03-18 05:17:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-18 05:17:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:37 proc_lsst.multi:170 [INFO] [multi] job_id 17194966 proc_lsst.multi INFO: [multi] job_id 17194966 2024-03-18 05:17:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-18 05:17:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:37 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17194966 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17194966 parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 17194966 parsl.executors.status_handling INFO: Allocated block ID 15 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15 2024-03-18 05:17:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 2024-03-18 05:17:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-18 05:17:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:38 proc_lsst.multi:170 [INFO] [multi] job_id 17194967 proc_lsst.multi INFO: [multi] job_id 17194967 2024-03-18 05:17:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-18 05:17:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:38 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17194967 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17194967 parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 17194967 parsl.executors.status_handling INFO: Allocated block ID 16 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16 2024-03-18 05:17:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 2024-03-18 05:17:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-18 05:17:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:38 proc_lsst.multi:170 [INFO] [multi] job_id 17194968 proc_lsst.multi INFO: [multi] job_id 17194968 2024-03-18 05:17:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-18 05:17:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:38 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17194968 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17194968 parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 17194968 parsl.executors.status_handling INFO: Allocated block ID 17 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17 2024-03-18 05:17:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 2024-03-18 05:17:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-18 05:17:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:38 proc_lsst.multi:170 [INFO] [multi] job_id 17194969 proc_lsst.multi INFO: [multi] job_id 17194969 2024-03-18 05:17:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-18 05:17:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:38 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17194969 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17194969 parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 17194969 parsl.executors.status_handling INFO: Allocated block ID 18 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18 2024-03-18 05:17:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 2024-03-18 05:17:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-18 05:17:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:39 proc_lsst.multi:170 [INFO] [multi] job_id 17194970 proc_lsst.multi INFO: [multi] job_id 17194970 2024-03-18 05:17:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-18 05:17:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:39 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17194970 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17194970 parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 17194970 parsl.executors.status_handling INFO: Allocated block ID 19 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19 2024-03-18 05:17:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 2024-03-18 05:17:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-18 05:17:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:39 proc_lsst.multi:170 [INFO] [multi] job_id 17194971 proc_lsst.multi INFO: [multi] job_id 17194971 2024-03-18 05:17:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-18 05:17:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:39 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17194971 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17194971 parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 17194971 parsl.executors.status_handling INFO: Allocated block ID 20 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20 2024-03-18 05:17:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 2024-03-18 05:17:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-18 05:17:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:40 proc_lsst.multi:170 [INFO] [multi] job_id 17194973 proc_lsst.multi INFO: [multi] job_id 17194973 2024-03-18 05:17:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 05:17:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:40 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17194973 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17194973 parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 17194973 parsl.executors.status_handling INFO: Allocated block ID 21 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21 2024-03-18 05:17:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 2024-03-18 05:17:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 05:17:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:40 proc_lsst.multi:170 [INFO] [multi] job_id 17194974 proc_lsst.multi INFO: [multi] job_id 17194974 2024-03-18 05:17:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-18 05:17:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:40 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17194974 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17194974 parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 17194974 parsl.executors.status_handling INFO: Allocated block ID 22 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22 2024-03-18 05:17:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 2024-03-18 05:17:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-18 05:17:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:40 proc_lsst.multi:170 [INFO] [multi] job_id 17194975 proc_lsst.multi INFO: [multi] job_id 17194975 2024-03-18 05:17:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-18 05:17:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:40 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17194975 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17194975 parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 17194975 parsl.executors.status_handling INFO: Allocated block ID 23 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23 2024-03-18 05:17:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 2024-03-18 05:17:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-18 05:17:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:40 proc_lsst.multi:170 [INFO] [multi] job_id 17194976 proc_lsst.multi INFO: [multi] job_id 17194976 2024-03-18 05:17:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-18 05:17:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:40 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17194976 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17194976 parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 17194976 parsl.executors.status_handling INFO: Allocated block ID 24 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24 2024-03-18 05:17:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 2024-03-18 05:17:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-18 05:17:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:40 proc_lsst.multi:170 [INFO] [multi] job_id 17194977 proc_lsst.multi INFO: [multi] job_id 17194977 2024-03-18 05:17:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-18 05:17:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:40 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17194977 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17194977 parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 17194977 parsl.executors.status_handling INFO: Allocated block ID 25 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25 2024-03-18 05:17:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 2024-03-18 05:17:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-18 05:17:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:40 proc_lsst.multi:170 [INFO] [multi] job_id 17194978 proc_lsst.multi INFO: [multi] job_id 17194978 2024-03-18 05:17:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-18 05:17:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:40 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17194978 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17194978 parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 17194978 parsl.executors.status_handling INFO: Allocated block ID 26 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26 2024-03-18 05:17:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 2024-03-18 05:17:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-18 05:17:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:41 proc_lsst.multi:170 [INFO] [multi] job_id 17194979 proc_lsst.multi INFO: [multi] job_id 17194979 2024-03-18 05:17:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-18 05:17:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:41 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17194979 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17194979 parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 17194979 parsl.executors.status_handling INFO: Allocated block ID 27 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27 2024-03-18 05:17:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 2024-03-18 05:17:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-18 05:17:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:41 proc_lsst.multi:170 [INFO] [multi] job_id 17194980 proc_lsst.multi INFO: [multi] job_id 17194980 2024-03-18 05:17:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-18 05:17:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:41 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17194980 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17194980 parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 17194980 parsl.executors.status_handling INFO: Allocated block ID 28 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28 2024-03-18 05:17:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 2024-03-18 05:17:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-18 05:17:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:41 proc_lsst.multi:170 [INFO] [multi] job_id 17194981 proc_lsst.multi INFO: [multi] job_id 17194981 2024-03-18 05:17:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-18 05:17:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:41 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17194981 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17194981 parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 17194981 parsl.executors.status_handling INFO: Allocated block ID 29 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29 2024-03-18 05:17:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 2024-03-18 05:17:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-18 05:17:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:41 proc_lsst.multi:170 [INFO] [multi] job_id 17194982 proc_lsst.multi INFO: [multi] job_id 17194982 2024-03-18 05:17:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-18 05:17:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:41 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17194982 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17194982 parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 17194982 parsl.executors.status_handling INFO: Allocated block ID 30 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30 2024-03-18 05:17:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 2024-03-18 05:17:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-18 05:17:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:41 proc_lsst.multi:170 [INFO] [multi] job_id 17194983 proc_lsst.multi INFO: [multi] job_id 17194983 2024-03-18 05:17:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:17:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:41 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17194983 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17194983 parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 17194983 parsl.executors.status_handling INFO: Allocated block ID 31 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-31 2024-03-18 05:17:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 2024-03-18 05:17:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:17:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:41 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-18 05:17:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:17:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:41 proc_lsst.multi:170 [INFO] [multi] job_id 17194984 proc_lsst.multi INFO: [multi] job_id 17194984 2024-03-18 05:17:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:17:41 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17194984 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17194984 parsl.executors.status_handling DEBUG: Launched block 31 on executor multi with job ID 17194984 parsl.executors.status_handling INFO: Allocated block ID 32 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-32 2024-03-18 05:17:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 2024-03-18 05:17:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:17:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:41 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:17:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:41 proc_lsst.multi:170 [INFO] [multi] job_id 17194985 proc_lsst.multi INFO: [multi] job_id 17194985 2024-03-18 05:17:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-18 05:17:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:17:41 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17194985 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17194985 parsl.executors.status_handling DEBUG: Launched block 32 on executor multi with job ID 17194985 parsl.executors.status_handling INFO: Allocated block ID 33 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-33 2024-03-18 05:17:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 2024-03-18 05:17:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:17:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:41 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-18 05:17:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:17:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:41 proc_lsst.multi:170 [INFO] [multi] job_id 17194986 proc_lsst.multi INFO: [multi] job_id 17194986 2024-03-18 05:17:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-18 05:17:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:17:41 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17194986 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17194986 parsl.executors.status_handling DEBUG: Launched block 33 on executor multi with job ID 17194986 parsl.executors.status_handling INFO: Allocated block ID 34 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-34 2024-03-18 05:17:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 2024-03-18 05:17:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:17:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:41 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-18 05:17:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:17:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:42 proc_lsst.multi:170 [INFO] [multi] job_id 17194987 proc_lsst.multi INFO: [multi] job_id 17194987 2024-03-18 05:17:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-18 05:17:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:17:42 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17194987 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17194987 parsl.executors.status_handling DEBUG: Launched block 34 on executor multi with job ID 17194987 parsl.executors.status_handling INFO: Allocated block ID 35 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-35 2024-03-18 05:17:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 2024-03-18 05:17:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:17:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:42 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-18 05:17:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:17:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:42 proc_lsst.multi:170 [INFO] [multi] job_id 17194988 proc_lsst.multi INFO: [multi] job_id 17194988 2024-03-18 05:17:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:17:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:17:42 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17194988 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17194988 parsl.executors.status_handling DEBUG: Launched block 35 on executor multi with job ID 17194988 parsl.executors.status_handling INFO: Allocated block ID 36 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-36 2024-03-18 05:17:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 2024-03-18 05:17:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:17:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:42 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:17:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:17:42 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-18 05:17:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:42 proc_lsst.multi:170 [INFO] [multi] job_id 17194989 proc_lsst.multi INFO: [multi] job_id 17194989 2024-03-18 05:17:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:42 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17194989 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17194989 parsl.executors.status_handling DEBUG: Launched block 36 on executor multi with job ID 17194989 parsl.executors.status_handling INFO: Allocated block ID 37 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-37 2024-03-18 05:17:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 2024-03-18 05:17:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:17:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:42 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:17:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:17:42 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:43 proc_lsst.multi:170 [INFO] [multi] job_id 17194990 proc_lsst.multi INFO: [multi] job_id 17194990 2024-03-18 05:17:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-18 05:17:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:43 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17194990 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17194990 parsl.executors.status_handling DEBUG: Launched block 37 on executor multi with job ID 17194990 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23151910176784 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 38, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 1/37 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 38 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 24 more blocks parsl.executors.status_handling INFO: Scaling out by 24 blocks parsl.executors.status_handling INFO: Allocated block ID 38 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-38 2024-03-18 05:17:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 2024-03-18 05:17:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:17:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:43 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:17:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:17:43 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-18 05:17:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:43 proc_lsst.multi:170 [INFO] [multi] job_id 17194991 proc_lsst.multi INFO: [multi] job_id 17194991 2024-03-18 05:17:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-18 05:17:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:43 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17194991 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17194991 parsl.executors.status_handling DEBUG: Launched block 38 on executor multi with job ID 17194991 parsl.executors.status_handling INFO: Allocated block ID 39 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-39 2024-03-18 05:17:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 2024-03-18 05:17:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:17:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:43 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:17:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:17:43 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-18 05:17:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:43 proc_lsst.multi:170 [INFO] [multi] job_id 17194992 proc_lsst.multi INFO: [multi] job_id 17194992 2024-03-18 05:17:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-18 05:17:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:43 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17194992 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17194992 parsl.executors.status_handling DEBUG: Launched block 39 on executor multi with job ID 17194992 parsl.executors.status_handling INFO: Allocated block ID 40 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-40 2024-03-18 05:17:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 2024-03-18 05:17:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:17:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:43 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:17:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:17:43 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-18 05:17:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:44 proc_lsst.multi:170 [INFO] [multi] job_id 17194993 proc_lsst.multi INFO: [multi] job_id 17194993 2024-03-18 05:17:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:17:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:44 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17194993 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17194993 parsl.executors.status_handling DEBUG: Launched block 40 on executor multi with job ID 17194993 parsl.executors.status_handling INFO: Allocated block ID 41 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-41 2024-03-18 05:17:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 2024-03-18 05:17:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:17:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:44 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:17:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:17:44 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:17:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:44 proc_lsst.multi:170 [INFO] [multi] job_id 17194994 proc_lsst.multi INFO: [multi] job_id 17194994 2024-03-18 05:17:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-18 05:17:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:44 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17194994 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17194994 parsl.executors.status_handling DEBUG: Launched block 41 on executor multi with job ID 17194994 parsl.executors.status_handling INFO: Allocated block ID 42 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-42 2024-03-18 05:17:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 2024-03-18 05:17:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:17:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:44 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:17:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:17:44 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-18 05:17:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:44 proc_lsst.multi:170 [INFO] [multi] job_id 17194995 proc_lsst.multi INFO: [multi] job_id 17194995 2024-03-18 05:17:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-18 05:17:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:44 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17194995 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17194995 parsl.executors.status_handling DEBUG: Launched block 42 on executor multi with job ID 17194995 parsl.executors.status_handling INFO: Allocated block ID 43 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-43 2024-03-18 05:17:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 2024-03-18 05:17:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:17:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:44 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:17:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:17:44 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-18 05:17:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:44 proc_lsst.multi:170 [INFO] [multi] job_id 17194996 proc_lsst.multi INFO: [multi] job_id 17194996 2024-03-18 05:17:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-18 05:17:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:44 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17194996 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17194996 parsl.executors.status_handling DEBUG: Launched block 43 on executor multi with job ID 17194996 parsl.executors.status_handling INFO: Allocated block ID 44 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-44 2024-03-18 05:17:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 2024-03-18 05:17:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:17:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:44 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:17:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:17:44 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-18 05:17:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:45 proc_lsst.multi:170 [INFO] [multi] job_id 17194997 proc_lsst.multi INFO: [multi] job_id 17194997 2024-03-18 05:17:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-18 05:17:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:45 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17194997 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17194997 parsl.executors.status_handling DEBUG: Launched block 44 on executor multi with job ID 17194997 parsl.executors.status_handling INFO: Allocated block ID 45 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-45 2024-03-18 05:17:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 2024-03-18 05:17:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:17:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:45 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:17:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:17:45 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-18 05:17:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:45 proc_lsst.multi:170 [INFO] [multi] job_id 17194998 proc_lsst.multi INFO: [multi] job_id 17194998 2024-03-18 05:17:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-18 05:17:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:45 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17194998 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17194998 parsl.executors.status_handling DEBUG: Launched block 45 on executor multi with job ID 17194998 parsl.executors.status_handling INFO: Allocated block ID 46 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-46 2024-03-18 05:17:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 2024-03-18 05:17:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:17:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:45 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:17:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:17:45 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-18 05:17:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9292cc4565be5050e102e240b5372537 with result from task 0 parsl.dataflow.dflow DEBUG: Task 0 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/1fa494c2-b018-4153-a728-b8c3510f795d_isrForCrosstalkSources_1040013_31.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/1fa494c2-b018-4153-a728-b8c3510f795d_isrForCrosstalkSources_1040013_31.stderr 2024-03-18 05:17:45 proc_lsst.multi:170 [INFO] [multi] job_id 17194999 proc_lsst.multi INFO: [multi] job_id 17194999 2024-03-18 05:17:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-18 05:17:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:45 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17194999 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17194999 parsl.executors.status_handling DEBUG: Launched block 46 on executor multi with job ID 17194999 parsl.executors.status_handling INFO: Allocated block ID 47 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-47 2024-03-18 05:17:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 2024-03-18 05:17:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:17:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:45 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:17:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:17:45 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-18 05:17:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c38a761148881ad983cba6300eda6f80 with result from task 1 parsl.dataflow.dflow DEBUG: Task 1 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/9528ae46-79a5-49f1-b416-9382a8dfd657_isrForCrosstalkSources_1039907_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/9528ae46-79a5-49f1-b416-9382a8dfd657_isrForCrosstalkSources_1039907_24.stderr 2024-03-18 05:17:45 proc_lsst.multi:170 [INFO] [multi] job_id 17195000 proc_lsst.multi INFO: [multi] job_id 17195000 2024-03-18 05:17:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-18 05:17:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:45 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17195000 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17195000 parsl.executors.status_handling DEBUG: Launched block 47 on executor multi with job ID 17195000 parsl.executors.status_handling INFO: Allocated block ID 48 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-48 2024-03-18 05:17:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 2024-03-18 05:17:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:17:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:45 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:17:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:17:45 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-18 05:17:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:46 proc_lsst.multi:170 [INFO] [multi] job_id 17195001 proc_lsst.multi INFO: [multi] job_id 17195001 2024-03-18 05:17:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-18 05:17:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:46 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17195001 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17195001 parsl.executors.status_handling DEBUG: Launched block 48 on executor multi with job ID 17195001 parsl.executors.status_handling INFO: Allocated block ID 49 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-49 2024-03-18 05:17:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 2024-03-18 05:17:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:17:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:46 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:17:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:17:46 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-18 05:17:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:46 proc_lsst.multi:170 [INFO] [multi] job_id 17195002 proc_lsst.multi INFO: [multi] job_id 17195002 2024-03-18 05:17:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-18 05:17:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:46 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17195002 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17195002 parsl.executors.status_handling DEBUG: Launched block 49 on executor multi with job ID 17195002 parsl.executors.status_handling INFO: Allocated block ID 50 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-50 2024-03-18 05:17:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 2024-03-18 05:17:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:17:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:46 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:17:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:17:46 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-18 05:17:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:46 proc_lsst.multi:170 [INFO] [multi] job_id 17195003 proc_lsst.multi INFO: [multi] job_id 17195003 2024-03-18 05:17:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-18 05:17:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:46 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17195003 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17195003 parsl.executors.status_handling DEBUG: Launched block 50 on executor multi with job ID 17195003 parsl.executors.status_handling INFO: Allocated block ID 51 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-51 2024-03-18 05:17:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 2024-03-18 05:17:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:17:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:46 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:17:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:17:46 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-18 05:17:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:47 proc_lsst.multi:170 [INFO] [multi] job_id 17195004 proc_lsst.multi INFO: [multi] job_id 17195004 2024-03-18 05:17:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-18 05:17:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:47 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17195004 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17195004 parsl.executors.status_handling DEBUG: Launched block 51 on executor multi with job ID 17195004 parsl.executors.status_handling INFO: Allocated block ID 52 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-52 2024-03-18 05:17:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 2024-03-18 05:17:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:17:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:17:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:17:47 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-18 05:17:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:47 proc_lsst.multi:170 [INFO] [multi] job_id 17195005 proc_lsst.multi INFO: [multi] job_id 17195005 2024-03-18 05:17:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-18 05:17:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:47 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17195005 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17195005 parsl.executors.status_handling DEBUG: Launched block 52 on executor multi with job ID 17195005 parsl.executors.status_handling INFO: Allocated block ID 53 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-53 2024-03-18 05:17:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 2024-03-18 05:17:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:17:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:17:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:17:47 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-18 05:17:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:47 proc_lsst.multi:170 [INFO] [multi] job_id 17195006 proc_lsst.multi INFO: [multi] job_id 17195006 2024-03-18 05:17:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-18 05:17:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:47 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17195006 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17195006 parsl.executors.status_handling DEBUG: Launched block 53 on executor multi with job ID 17195006 parsl.executors.status_handling INFO: Allocated block ID 54 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-54 2024-03-18 05:17:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 2024-03-18 05:17:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:17:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:17:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:17:47 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-18 05:17:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:47 proc_lsst.multi:170 [INFO] [multi] job_id 17195007 proc_lsst.multi INFO: [multi] job_id 17195007 2024-03-18 05:17:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-18 05:17:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:47 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17195007 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17195007 parsl.executors.status_handling DEBUG: Launched block 54 on executor multi with job ID 17195007 parsl.executors.status_handling INFO: Allocated block ID 55 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-55 2024-03-18 05:17:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 2024-03-18 05:17:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:17:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:17:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:17:47 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-18 05:17:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:47 proc_lsst.multi:170 [INFO] [multi] job_id 17195008 proc_lsst.multi INFO: [multi] job_id 17195008 2024-03-18 05:17:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 05:17:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:47 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17195008 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17195008 parsl.executors.status_handling DEBUG: Launched block 55 on executor multi with job ID 17195008 parsl.executors.status_handling INFO: Allocated block ID 56 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-56 2024-03-18 05:17:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 2024-03-18 05:17:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:17:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:17:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:17:47 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 05:17:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:47 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-18 05:17:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 05:17:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:48 proc_lsst.multi:170 [INFO] [multi] job_id 17195009 proc_lsst.multi INFO: [multi] job_id 17195009 2024-03-18 05:17:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 05:17:48 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17195009 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17195009 parsl.executors.status_handling DEBUG: Launched block 56 on executor multi with job ID 17195009 parsl.executors.status_handling INFO: Allocated block ID 57 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-57 2024-03-18 05:17:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 2024-03-18 05:17:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:17:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:48 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:17:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:17:48 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 05:17:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:48 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 05:17:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:48 proc_lsst.multi:170 [INFO] [multi] job_id 17195010 proc_lsst.multi INFO: [multi] job_id 17195010 2024-03-18 05:17:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-18 05:17:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 05:17:48 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17195010 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17195010 parsl.executors.status_handling DEBUG: Launched block 57 on executor multi with job ID 17195010 parsl.executors.status_handling INFO: Allocated block ID 58 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-58 2024-03-18 05:17:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 2024-03-18 05:17:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:17:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:48 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:17:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:17:48 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 05:17:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:48 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-18 05:17:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 05:17:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:48 proc_lsst.multi:170 [INFO] [multi] job_id 17195011 proc_lsst.multi INFO: [multi] job_id 17195011 2024-03-18 05:17:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-18 05:17:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 05:17:48 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17195011 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17195011 parsl.executors.status_handling DEBUG: Launched block 58 on executor multi with job ID 17195011 parsl.executors.status_handling INFO: Allocated block ID 59 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-59 2024-03-18 05:17:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 2024-03-18 05:17:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:17:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:48 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:17:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:17:48 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 05:17:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:48 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-18 05:17:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 05:17:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:49 proc_lsst.multi:170 [INFO] [multi] job_id 17195012 proc_lsst.multi INFO: [multi] job_id 17195012 2024-03-18 05:17:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-18 05:17:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 05:17:49 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17195012 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17195012 parsl.executors.status_handling DEBUG: Launched block 59 on executor multi with job ID 17195012 parsl.executors.status_handling INFO: Allocated block ID 60 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-60 2024-03-18 05:17:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 2024-03-18 05:17:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:17:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:49 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:17:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:17:49 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 05:17:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:49 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-18 05:17:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 05:17:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:49 proc_lsst.multi:170 [INFO] [multi] job_id 17195013 proc_lsst.multi INFO: [multi] job_id 17195013 2024-03-18 05:17:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:17:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 05:17:49 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17195013 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17195013 parsl.executors.status_handling DEBUG: Launched block 60 on executor multi with job ID 17195013 parsl.executors.status_handling INFO: Allocated block ID 61 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-61 2024-03-18 05:17:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 2024-03-18 05:17:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 05:17:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 05:17:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 05:17:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 05:17:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 05:17:49 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:17:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 05:17:49 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 05:17:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 05:17:49 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 05:17:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 05:17:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 05:17:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54090 --result_port=54515 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 05:17:49 proc_lsst.multi:170 [INFO] [multi] job_id 17195014 proc_lsst.multi INFO: [multi] job_id 17195014 2024-03-18 05:17:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-18 05:17:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 05:17:49 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17195014 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17195014 parsl.executors.status_handling DEBUG: Launched block 61 on executor multi with job ID 17195014 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23151910176784 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 246 parsl.dataflow.strategy DEBUG: Executor multi has 246 active tasks, 1/61 running/pending blocks, and 22 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23151910176784 parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ccd79b7edd1db35380c92ce7852838f with result from task 3 parsl.dataflow.dflow DEBUG: Task 3 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/4f1fb978-903d-4692-89a3-0765cbaa6167_isrForCrosstalkSources_1039909_18.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/4f1fb978-903d-4692-89a3-0765cbaa6167_isrForCrosstalkSources_1039909_18.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 245 parsl.dataflow.strategy DEBUG: Executor multi has 245 active tasks, 1/61 running/pending blocks, and 56 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23151910176784 parsl.dataflow.dflow INFO: Task 2 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f880ab4e16b72592f6bdd8925deb2ead with result from task 2 parsl.dataflow.dflow DEBUG: Task 2 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/2219ea56-9870-4069-b142-9844b9e7c4b7_isrForCrosstalkSources_1039907_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/2219ea56-9870-4069-b142-9844b9e7c4b7_isrForCrosstalkSources_1039907_61.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 244 parsl.dataflow.strategy DEBUG: Executor multi has 244 active tasks, 1/61 running/pending blocks, and 57 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23151910176784 2024-03-18 05:18:04 proc_lsst.multi:146 [INFO] found job 53289 in provider local proc_lsst.multi INFO: found job 53289 in provider local 2024-03-18 05:18:04 proc_lsst.multi:146 [INFO] found job 17194953 in provider astro proc_lsst.multi INFO: found job 17194953 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.dataflow.dflow INFO: Task 57 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07b183ba6cd63b799594c7c2fbaa3170 with result from task 57 parsl.dataflow.dflow DEBUG: Task 57 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/3cff5095-9540-489d-9d2e-65d0a295bf6c_isrForCrosstalkSources_1039909_55.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/3cff5095-9540-489d-9d2e-65d0a295bf6c_isrForCrosstalkSources_1039909_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:05 proc_lsst.multi:146 [INFO] found job 17194954 in provider astro proc_lsst.multi INFO: found job 17194954 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:06 proc_lsst.multi:146 [INFO] found job 17194955 in provider astro proc_lsst.multi INFO: found job 17194955 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.dataflow.dflow INFO: Task 59 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6103710cf04990dd35c081090f6e395 with result from task 59 parsl.dataflow.dflow DEBUG: Task 59 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/b18cad74-fd87-4fba-a08a-8d063c662869_isrForCrosstalkSources_1039909_17.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/b18cad74-fd87-4fba-a08a-8d063c662869_isrForCrosstalkSources_1039909_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:06 proc_lsst.multi:146 [INFO] found job 17194956 in provider astro proc_lsst.multi INFO: found job 17194956 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:07 proc_lsst.multi:146 [INFO] found job 17194957 in provider astro proc_lsst.multi INFO: found job 17194957 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:07 proc_lsst.multi:146 [INFO] found job 17194958 in provider astro proc_lsst.multi INFO: found job 17194958 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:07 proc_lsst.multi:146 [INFO] found job 17194959 in provider astro proc_lsst.multi INFO: found job 17194959 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:08 proc_lsst.multi:146 [INFO] found job 17194960 in provider astro proc_lsst.multi INFO: found job 17194960 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:08 proc_lsst.multi:146 [INFO] found job 17194961 in provider astro proc_lsst.multi INFO: found job 17194961 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:09 proc_lsst.multi:146 [INFO] found job 17194962 in provider astro proc_lsst.multi INFO: found job 17194962 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:09 proc_lsst.multi:146 [INFO] found job 17194963 in provider astro proc_lsst.multi INFO: found job 17194963 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:10 proc_lsst.multi:146 [INFO] found job 17194964 in provider astro proc_lsst.multi INFO: found job 17194964 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:11 proc_lsst.multi:146 [INFO] found job 17194965 in provider astro proc_lsst.multi INFO: found job 17194965 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:11 proc_lsst.multi:146 [INFO] found job 17194966 in provider astro proc_lsst.multi INFO: found job 17194966 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:11 proc_lsst.multi:146 [INFO] found job 17194967 in provider astro proc_lsst.multi INFO: found job 17194967 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:12 proc_lsst.multi:146 [INFO] found job 17194968 in provider astro proc_lsst.multi INFO: found job 17194968 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:12 proc_lsst.multi:146 [INFO] found job 17194969 in provider astro proc_lsst.multi INFO: found job 17194969 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:13 proc_lsst.multi:146 [INFO] found job 17194970 in provider astro proc_lsst.multi INFO: found job 17194970 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:13 proc_lsst.multi:146 [INFO] found job 17194971 in provider astro proc_lsst.multi INFO: found job 17194971 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:13 proc_lsst.multi:146 [INFO] found job 17194973 in provider astro proc_lsst.multi INFO: found job 17194973 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:14 proc_lsst.multi:146 [INFO] found job 17194974 in provider astro proc_lsst.multi INFO: found job 17194974 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.dataflow.dflow INFO: Task 62 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d012688b2a8397caa8de1458bf38ae3a with result from task 62 parsl.dataflow.dflow DEBUG: Task 62 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/5044c9ba-588d-4a49-beee-2b9539748c13_isrForCrosstalkSources_1039909_56.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/5044c9ba-588d-4a49-beee-2b9539748c13_isrForCrosstalkSources_1039909_56.stderr parsl.dataflow.dflow INFO: Task 61 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 079f90f00cd4f92e9e6070d19ea181f4 with result from task 61 parsl.dataflow.dflow DEBUG: Task 61 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/fc7fa6ef-9610-414a-b44a-6195ac99afde_isrForCrosstalkSources_1039909_15.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/fc7fa6ef-9610-414a-b44a-6195ac99afde_isrForCrosstalkSources_1039909_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:14 proc_lsst.multi:146 [INFO] found job 17194975 in provider astro proc_lsst.multi INFO: found job 17194975 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.dataflow.dflow INFO: Task 35 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af2103163dfe4346a62a96cb12d3e90c with result from task 35 parsl.dataflow.dflow DEBUG: Task 35 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/248f92e0-cb6c-4bd2-9ee8-9e8e690351a5_isrForCrosstalkSources_1040013_8.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/248f92e0-cb6c-4bd2-9ee8-9e8e690351a5_isrForCrosstalkSources_1040013_8.stderr parsl.dataflow.dflow INFO: Task 29 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42e560c82fb993c4d5e72176a5c01bbb with result from task 29 parsl.dataflow.dflow DEBUG: Task 29 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/b2bf63ba-1d07-4a3c-9bdf-99525dc0f827_isrForCrosstalkSources_1039907_7.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/b2bf63ba-1d07-4a3c-9bdf-99525dc0f827_isrForCrosstalkSources_1039907_7.stderr parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5dc74189f1dfd48677cfd0fb0509733f with result from task 30 parsl.dataflow.dflow DEBUG: Task 30 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/aa03bb71-bc2e-41bc-8c9d-8d6d0dbcde2b_isrForCrosstalkSources_1039909_51.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/aa03bb71-bc2e-41bc-8c9d-8d6d0dbcde2b_isrForCrosstalkSources_1039909_51.stderr parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8dfad9e5fbbe54ad7eecb3e87a1f431d with result from task 33 parsl.dataflow.dflow DEBUG: Task 33 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/c28912f7-8aed-4d99-8f88-0416951a5142_isrForCrosstalkSources_1039907_31.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/c28912f7-8aed-4d99-8f88-0416951a5142_isrForCrosstalkSources_1039907_31.stderr parsl.dataflow.dflow INFO: Task 32 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82d2cd3ab6ad0b28bc1b121ea3336998 with result from task 32 parsl.dataflow.dflow DEBUG: Task 32 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/885f8a7e-4636-46a4-879d-697bde2544f8_isrForCrosstalkSources_1039908_44.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/885f8a7e-4636-46a4-879d-697bde2544f8_isrForCrosstalkSources_1039908_44.stderr parsl.dataflow.dflow INFO: Task 26 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce571dcef8c0f79f74ff2fce2ab12645 with result from task 26 parsl.dataflow.dflow DEBUG: Task 26 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/606e4ae1-82cd-4a74-9b53-9deaa7e98678_isrForCrosstalkSources_1040013_5.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/606e4ae1-82cd-4a74-9b53-9deaa7e98678_isrForCrosstalkSources_1040013_5.stderr parsl.dataflow.dflow INFO: Task 25 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e382376b46e3dc4f2867cc83b5a92cb with result from task 25 parsl.dataflow.dflow DEBUG: Task 25 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/63034363-3d3a-44ef-8a2f-f69b2f3f429c_isrForCrosstalkSources_1040013_30.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/63034363-3d3a-44ef-8a2f-f69b2f3f429c_isrForCrosstalkSources_1040013_30.stderr parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8f3507c265ae688fb2d83099833965a with result from task 27 parsl.dataflow.dflow DEBUG: Task 27 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/314cdf80-2ea2-41f7-8a46-75337853466c_isrForCrosstalkSources_1040013_15.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/314cdf80-2ea2-41f7-8a46-75337853466c_isrForCrosstalkSources_1040013_15.stderr parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 600802e602cc5e6ccc7731cb7520d42c with result from task 24 parsl.dataflow.dflow DEBUG: Task 24 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/f72fa67b-c3be-4709-9608-4d48ef0aada4_isrForCrosstalkSources_1039908_61.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/f72fa67b-c3be-4709-9608-4d48ef0aada4_isrForCrosstalkSources_1039908_61.stderr parsl.dataflow.dflow INFO: Task 28 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6879595d3b67f4dcdadcdb3f6c4cd226 with result from task 28 parsl.dataflow.dflow DEBUG: Task 28 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/ac7ef3a9-1e82-49cc-9dd4-39277772bfa6_isrForCrosstalkSources_1039907_41.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/ac7ef3a9-1e82-49cc-9dd4-39277772bfa6_isrForCrosstalkSources_1039907_41.stderr parsl.dataflow.dflow INFO: Task 40 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8cc9edd83f1818866bda214f23d08c7 with result from task 40 parsl.dataflow.dflow DEBUG: Task 40 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/45ced5c6-7552-4f53-943b-be8422b9bb9a_isrForCrosstalkSources_1039908_51.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/45ced5c6-7552-4f53-943b-be8422b9bb9a_isrForCrosstalkSources_1039908_51.stderr parsl.dataflow.dflow INFO: Task 58 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d6e252ee29b906267ab5fd420d1bef5d with result from task 58 parsl.dataflow.dflow DEBUG: Task 58 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/9185d49c-be8e-41b9-8dca-bbad79557879_isrForCrosstalkSources_1039908_1.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/9185d49c-be8e-41b9-8dca-bbad79557879_isrForCrosstalkSources_1039908_1.stderr parsl.dataflow.dflow INFO: Task 45 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e6995742ed05cfc561c24e240ea10c0 with result from task 45 parsl.dataflow.dflow DEBUG: Task 45 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/a9e8ce2a-4c44-4e0f-ab69-7ad3e14423a7_isrForCrosstalkSources_1039907_51.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/a9e8ce2a-4c44-4e0f-ab69-7ad3e14423a7_isrForCrosstalkSources_1039907_51.stderr parsl.dataflow.dflow INFO: Task 39 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8af6bfbc61a40d24a8ca7fe2b0f44939 with result from task 39 parsl.dataflow.dflow DEBUG: Task 39 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/0e3f3c7b-ad64-4068-b0c4-e6cdf50a7cff_isrForCrosstalkSources_1040013_1.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/0e3f3c7b-ad64-4068-b0c4-e6cdf50a7cff_isrForCrosstalkSources_1040013_1.stderr parsl.dataflow.dflow INFO: Task 44 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c23d06642cff937cc4aa5fa040988488 with result from task 44 parsl.dataflow.dflow DEBUG: Task 44 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/0b6d8f4c-1bab-4b57-908c-403f2bdd998c_isrForCrosstalkSources_1039907_35.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/0b6d8f4c-1bab-4b57-908c-403f2bdd998c_isrForCrosstalkSources_1039907_35.stderr parsl.dataflow.dflow INFO: Task 37 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 013d9fcce33d6a0e2eda9d0ccd7951cf with result from task 37 parsl.dataflow.dflow DEBUG: Task 37 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/5228935c-738f-44e1-bd9f-106128f27123_isrForCrosstalkSources_1040013_33.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/5228935c-738f-44e1-bd9f-106128f27123_isrForCrosstalkSources_1040013_33.stderr parsl.dataflow.dflow INFO: Task 43 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8922ebf3803a40ff2bde8096d0341074 with result from task 43 parsl.dataflow.dflow DEBUG: Task 43 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/5ad1a495-3f86-454c-a8f4-2b9ac20635cf_isrForCrosstalkSources_1039908_16.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/5ad1a495-3f86-454c-a8f4-2b9ac20635cf_isrForCrosstalkSources_1039908_16.stderr parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55b4764c6edf56cb0ecec4133a7caa25 with result from task 36 parsl.dataflow.dflow DEBUG: Task 36 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/601b1e3e-a88e-476b-ba08-8234969cd0ce_isrForCrosstalkSources_1039907_48.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/601b1e3e-a88e-476b-ba08-8234969cd0ce_isrForCrosstalkSources_1039907_48.stderr parsl.dataflow.dflow INFO: Task 42 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f1b8fe0516d431e3698b2d909eefae1 with result from task 42 parsl.dataflow.dflow DEBUG: Task 42 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/3640c35c-555c-4a84-a21d-06da2f35cc19_isrForCrosstalkSources_1039909_16.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/3640c35c-555c-4a84-a21d-06da2f35cc19_isrForCrosstalkSources_1039909_16.stderr parsl.dataflow.dflow INFO: Task 34 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68c1c0992b55eb0a6a84dceeb82a74a3 with result from task 34 parsl.dataflow.dflow DEBUG: Task 34 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/804b6a4c-2c7d-41f6-a52a-ab30f493f66b_isrForCrosstalkSources_1039907_6.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/804b6a4c-2c7d-41f6-a52a-ab30f493f66b_isrForCrosstalkSources_1039907_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:14 proc_lsst.multi:146 [INFO] found job 17194976 in provider astro proc_lsst.multi INFO: found job 17194976 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.dataflow.dflow INFO: Task 60 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e854d16b686f8d755bb8065b68e8611 with result from task 60 parsl.dataflow.dflow DEBUG: Task 60 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/388e8d06-56d7-4af3-8a38-c3c0cf53a2c5_isrForCrosstalkSources_1039908_50.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/388e8d06-56d7-4af3-8a38-c3c0cf53a2c5_isrForCrosstalkSources_1039908_50.stderr parsl.dataflow.dflow INFO: Task 46 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9b65582e9301580eb6d4038ca0ed4ce with result from task 46 parsl.dataflow.dflow DEBUG: Task 46 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/bdf18bee-c3ad-4cb5-992c-27bbabb33272_isrForCrosstalkSources_1039908_48.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/bdf18bee-c3ad-4cb5-992c-27bbabb33272_isrForCrosstalkSources_1039908_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:15 proc_lsst.multi:146 [INFO] found job 17194977 in provider astro proc_lsst.multi INFO: found job 17194977 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.dataflow.dflow INFO: Task 38 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79a695ba1e1cda8c33bccca65a537067 with result from task 38 parsl.dataflow.dflow DEBUG: Task 38 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/a0dc0534-f384-4501-8e5c-0b4ee3823213_isrForCrosstalkSources_1039908_43.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/a0dc0534-f384-4501-8e5c-0b4ee3823213_isrForCrosstalkSources_1039908_43.stderr parsl.dataflow.dflow INFO: Task 31 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6073278a83eb0878c1debbe49b263aee with result from task 31 parsl.dataflow.dflow DEBUG: Task 31 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/f3cf710f-a847-4e0d-ab9e-311bafe29591_isrForCrosstalkSources_1039909_29.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/f3cf710f-a847-4e0d-ab9e-311bafe29591_isrForCrosstalkSources_1039909_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:15 proc_lsst.multi:146 [INFO] found job 17194978 in provider astro proc_lsst.multi INFO: found job 17194978 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.dataflow.dflow INFO: Task 41 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d59065c3fd874dacb46a747e1af897c with result from task 41 parsl.dataflow.dflow DEBUG: Task 41 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/879ca815-0ad7-437c-9398-61b0b5203e44_isrForCrosstalkSources_1039907_62.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/879ca815-0ad7-437c-9398-61b0b5203e44_isrForCrosstalkSources_1039907_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:15 proc_lsst.multi:146 [INFO] found job 17194979 in provider astro proc_lsst.multi INFO: found job 17194979 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:16 proc_lsst.multi:146 [INFO] found job 17194980 in provider astro proc_lsst.multi INFO: found job 17194980 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:16 proc_lsst.multi:146 [INFO] found job 17194981 in provider astro proc_lsst.multi INFO: found job 17194981 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:17 proc_lsst.multi:146 [INFO] found job 17194982 in provider astro proc_lsst.multi INFO: found job 17194982 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.dataflow.dflow INFO: Task 56 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f3adc7781b3fb4031ac2d1932920d88 with result from task 56 parsl.dataflow.dflow DEBUG: Task 56 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/208e2b07-de2c-4d6d-8d77-db0a1a78cef1_isrForCrosstalkSources_1039909_32.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/208e2b07-de2c-4d6d-8d77-db0a1a78cef1_isrForCrosstalkSources_1039909_32.stderr parsl.dataflow.dflow INFO: Task 50 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 74db28982ab2f1716d00e1b62357cfe2 with result from task 50 parsl.dataflow.dflow DEBUG: Task 50 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/6acb786b-61a6-4ff0-a145-d44960d8a8b7_isrForCrosstalkSources_1039909_14.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/6acb786b-61a6-4ff0-a145-d44960d8a8b7_isrForCrosstalkSources_1039909_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:17 proc_lsst.multi:146 [INFO] found job 17194983 in provider astro proc_lsst.multi INFO: found job 17194983 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.dataflow.dflow INFO: Task 54 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8263710854b0a781293eb6c5817e171b with result from task 54 parsl.dataflow.dflow DEBUG: Task 54 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/6b3b8b2a-98da-4f75-8d66-aa4d35013c9a_isrForCrosstalkSources_1039908_21.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/6b3b8b2a-98da-4f75-8d66-aa4d35013c9a_isrForCrosstalkSources_1039908_21.stderr parsl.dataflow.dflow INFO: Task 49 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61249d994f4727273d42860f6fb05e29 with result from task 49 parsl.dataflow.dflow DEBUG: Task 49 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/632d4347-3a68-4961-97c5-06d431910130_isrForCrosstalkSources_1039907_11.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/632d4347-3a68-4961-97c5-06d431910130_isrForCrosstalkSources_1039907_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:18 proc_lsst.multi:146 [INFO] found job 17194984 in provider escience_rtx6k proc_lsst.multi INFO: found job 17194984 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194984,17194985,17194986,17194987,17194988' parsl.dataflow.dflow INFO: Task 47 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93b39e929190f7832f4f02dc6268528b with result from task 47 parsl.dataflow.dflow DEBUG: Task 47 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/4781ae54-5d1c-4808-b481-f9caed6d35fc_isrForCrosstalkSources_1039909_7.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/4781ae54-5d1c-4808-b481-f9caed6d35fc_isrForCrosstalkSources_1039909_7.stderr parsl.dataflow.dflow INFO: Task 53 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50c8e20f322f782401aa65c2fd34adf9 with result from task 53 parsl.dataflow.dflow DEBUG: Task 53 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/332509ab-57e7-4286-ad93-422efdd41e1c_isrForCrosstalkSources_1040013_2.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/332509ab-57e7-4286-ad93-422efdd41e1c_isrForCrosstalkSources_1040013_2.stderr parsl.dataflow.dflow INFO: Task 17 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7fd1600935310c4c3631aa4683125599 with result from task 17 parsl.dataflow.dflow DEBUG: Task 17 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/ad1761bd-6667-4577-a517-868c8ac9023d_isrForCrosstalkSources_1039908_19.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/ad1761bd-6667-4577-a517-868c8ac9023d_isrForCrosstalkSources_1039908_19.stderr parsl.dataflow.dflow INFO: Task 16 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6156504d21e6482f00236aac4b79e139 with result from task 16 parsl.dataflow.dflow DEBUG: Task 16 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/fcca103c-d5fd-411e-bc1c-404542d6aeef_isrForCrosstalkSources_1039908_27.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/fcca103c-d5fd-411e-bc1c-404542d6aeef_isrForCrosstalkSources_1039908_27.stderr parsl.dataflow.dflow INFO: Task 11 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3331f5a2807ffd60f9c9c22d17318e2 with result from task 11 parsl.dataflow.dflow DEBUG: Task 11 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/147d14b6-4d37-492d-b97a-ee8c6b68d0a2_isrForCrosstalkSources_1039907_20.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/147d14b6-4d37-492d-b97a-ee8c6b68d0a2_isrForCrosstalkSources_1039907_20.stderr parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4772b4b1f835b556be5995356eeef030 with result from task 13 parsl.dataflow.dflow DEBUG: Task 13 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/fbdc779d-4994-44e8-b4ed-9b49040fcc9c_isrForCrosstalkSources_1039907_8.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/fbdc779d-4994-44e8-b4ed-9b49040fcc9c_isrForCrosstalkSources_1039907_8.stderr parsl.dataflow.dflow INFO: Task 4 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a473a5c2e95fe5209bb247a1ce6a0b69 with result from task 4 parsl.dataflow.dflow DEBUG: Task 4 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/2faf5afa-8e6f-42e2-9ce3-b2c655f7ceef_isrForCrosstalkSources_1040013_39.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/2faf5afa-8e6f-42e2-9ce3-b2c655f7ceef_isrForCrosstalkSources_1040013_39.stderr parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43cb28c3303b66d736b045360ac3d1f6 with result from task 15 parsl.dataflow.dflow DEBUG: Task 15 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/9de6dfe6-1773-4e88-b2e6-05040f8b01b5_isrForCrosstalkSources_1039909_26.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/9de6dfe6-1773-4e88-b2e6-05040f8b01b5_isrForCrosstalkSources_1039909_26.stderr parsl.dataflow.dflow INFO: Task 10 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06fa38e294e42cd4135f8f53ec8b25f3 with result from task 10 parsl.dataflow.dflow DEBUG: Task 10 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/437a8950-be02-479f-9641-2e2860915109_isrForCrosstalkSources_1039909_27.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/437a8950-be02-479f-9641-2e2860915109_isrForCrosstalkSources_1039909_27.stderr parsl.dataflow.dflow INFO: Task 7 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69aeaa2aa598177a09ed054d243f33a3 with result from task 7 parsl.dataflow.dflow DEBUG: Task 7 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/b189a951-59ae-4ff1-a739-bfae62bea25a_isrForCrosstalkSources_1039907_25.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/b189a951-59ae-4ff1-a739-bfae62bea25a_isrForCrosstalkSources_1039907_25.stderr parsl.dataflow.dflow INFO: Task 8 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f569bc7214c87604ddcade1b4b54eb2b with result from task 8 parsl.dataflow.dflow DEBUG: Task 8 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/e0f54475-421a-4f57-8943-5bb7b7096520_isrForCrosstalkSources_1039907_19.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/e0f54475-421a-4f57-8943-5bb7b7096520_isrForCrosstalkSources_1039907_19.stderr parsl.dataflow.dflow INFO: Task 51 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08969a243ec3896b0a2975133de6d9b8 with result from task 51 parsl.dataflow.dflow DEBUG: Task 51 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/4405fec8-2519-4f20-88bb-5425e09f3144_isrForCrosstalkSources_1039909_31.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/4405fec8-2519-4f20-88bb-5425e09f3144_isrForCrosstalkSources_1039909_31.stderr parsl.dataflow.dflow INFO: Task 48 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 275d3bd13279ea8c9f9eb3d8d404f0fe with result from task 48 parsl.dataflow.dflow DEBUG: Task 48 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/13ae3c66-ad45-4e99-924b-98b271e3bd82_isrForCrosstalkSources_1040013_44.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/13ae3c66-ad45-4e99-924b-98b271e3bd82_isrForCrosstalkSources_1040013_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17194984 R 17194985 R 17194986 R 17194987 R 17194988 R parsl.providers.slurm.slurm DEBUG: Updating job 17194984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194988 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:18 proc_lsst.multi:146 [INFO] found job 17194985 in provider escience_rtx6k proc_lsst.multi INFO: found job 17194985 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194984,17194985,17194986,17194987,17194988' parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 024e87170a1c4c78c3952ceffaee86e0 with result from task 6 parsl.dataflow.dflow DEBUG: Task 6 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/4909af3c-4287-45c4-afcc-5d880e43d3d8_isrForCrosstalkSources_1039908_18.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/4909af3c-4287-45c4-afcc-5d880e43d3d8_isrForCrosstalkSources_1039908_18.stderr parsl.dataflow.dflow INFO: Task 55 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22d529b997d13c64211d2690a4b53b5e with result from task 55 parsl.dataflow.dflow DEBUG: Task 55 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/60138b96-53c3-4598-a106-51615c6fd984_isrForCrosstalkSources_1039908_28.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/60138b96-53c3-4598-a106-51615c6fd984_isrForCrosstalkSources_1039908_28.stderr parsl.dataflow.dflow INFO: Task 52 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f659561cd09941ff5a3fb762ecf3ea7 with result from task 52 parsl.dataflow.dflow DEBUG: Task 52 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/fba9cb43-a684-4ef1-83c9-030768aef68e_isrForCrosstalkSources_1039907_42.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/fba9cb43-a684-4ef1-83c9-030768aef68e_isrForCrosstalkSources_1039907_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17194984 R 17194985 R 17194986 R 17194987 R 17194988 R parsl.providers.slurm.slurm DEBUG: Updating job 17194984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194988 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:18 proc_lsst.multi:146 [INFO] found job 17194986 in provider escience_rtx6k proc_lsst.multi INFO: found job 17194986 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194984,17194985,17194986,17194987,17194988' parsl.dataflow.dflow INFO: Task 20 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a774e06f04192d4311eebc1b7319046c with result from task 20 parsl.dataflow.dflow DEBUG: Task 20 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/facd5b4b-b4d8-4f12-896b-4c4e98d0a755_isrForCrosstalkSources_1040013_35.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/facd5b4b-b4d8-4f12-896b-4c4e98d0a755_isrForCrosstalkSources_1040013_35.stderr parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 736f90b928f2ce68bc8dd5a25a78a57e with result from task 21 parsl.dataflow.dflow DEBUG: Task 21 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/a92de5de-16ba-491c-a9a7-3db976b12b17_isrForCrosstalkSources_1039908_12.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/a92de5de-16ba-491c-a9a7-3db976b12b17_isrForCrosstalkSources_1039908_12.stderr parsl.dataflow.dflow INFO: Task 22 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e48ee2d04a65363b05977b3e8040964 with result from task 22 parsl.dataflow.dflow DEBUG: Task 22 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/98679fe2-2616-4d7d-ba76-79919bca4c17_isrForCrosstalkSources_1040013_41.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/98679fe2-2616-4d7d-ba76-79919bca4c17_isrForCrosstalkSources_1040013_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17194984 R 17194985 R 17194986 R 17194987 R 17194988 R parsl.providers.slurm.slurm DEBUG: Updating job 17194984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194988 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:19 proc_lsst.multi:146 [INFO] found job 17194987 in provider escience_rtx6k proc_lsst.multi INFO: found job 17194987 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194984,17194985,17194986,17194987,17194988' parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a23bdc851c561a0b4cedc7abe1886f6 with result from task 12 parsl.dataflow.dflow DEBUG: Task 12 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/db465d62-ae44-425e-a290-8ef45f54d0ec_isrForCrosstalkSources_1040013_47.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/db465d62-ae44-425e-a290-8ef45f54d0ec_isrForCrosstalkSources_1040013_47.stderr parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e69126f886d6a3ba30240d35514f05e0 with result from task 18 parsl.dataflow.dflow DEBUG: Task 18 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/afaa291e-1057-442e-93b2-c9fc5f7b1e86_isrForCrosstalkSources_1039908_31.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/afaa291e-1057-442e-93b2-c9fc5f7b1e86_isrForCrosstalkSources_1039908_31.stderr parsl.dataflow.dflow INFO: Task 19 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4f3a6211284eaae1894516a5109da6a with result from task 19 parsl.dataflow.dflow DEBUG: Task 19 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/a44e0614-264b-4444-98e2-825393ec13da_isrForCrosstalkSources_1039909_52.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/a44e0614-264b-4444-98e2-825393ec13da_isrForCrosstalkSources_1039909_52.stderr parsl.dataflow.dflow INFO: Task 5 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b296462d428d40a3f30280757a6e306c with result from task 5 parsl.dataflow.dflow DEBUG: Task 5 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/801f7551-0244-49fb-a550-c375ee167613_isrForCrosstalkSources_1039909_30.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/801f7551-0244-49fb-a550-c375ee167613_isrForCrosstalkSources_1039909_30.stderr parsl.dataflow.dflow INFO: Task 23 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e48562e71396a9397d7d84984e3d525 with result from task 23 parsl.dataflow.dflow DEBUG: Task 23 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/a1526ffd-bf44-44d5-afd1-3bf58c94b156_isrForCrosstalkSources_1039908_47.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/a1526ffd-bf44-44d5-afd1-3bf58c94b156_isrForCrosstalkSources_1039908_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17194984 R 17194985 R 17194986 R 17194987 R 17194988 R parsl.providers.slurm.slurm DEBUG: Updating job 17194984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194988 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:19 proc_lsst.multi:146 [INFO] found job 17194988 in provider escience_rtx6k proc_lsst.multi INFO: found job 17194988 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194984,17194985,17194986,17194987,17194988' parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43211b29dfe3a352332abdab496d8f26 with result from task 9 parsl.dataflow.dflow DEBUG: Task 9 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/08b32927-46a3-4ce8-8d95-4750a71160cf_isrForCrosstalkSources_1039908_55.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/08b32927-46a3-4ce8-8d95-4750a71160cf_isrForCrosstalkSources_1039908_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17194984 R 17194985 R 17194986 R 17194987 R 17194988 R parsl.providers.slurm.slurm DEBUG: Updating job 17194984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194988 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:20 proc_lsst.multi:146 [INFO] found job 17194989 in provider escience_a40 proc_lsst.multi INFO: found job 17194989 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8879cba6421f639d0362f31329178330 with result from task 14 parsl.dataflow.dflow DEBUG: Task 14 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/0e3920fc-fb96-4fcd-a8df-08dad46515b1_isrForCrosstalkSources_1039907_54.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/0e3920fc-fb96-4fcd-a8df-08dad46515b1_isrForCrosstalkSources_1039907_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:20 proc_lsst.multi:146 [INFO] found job 17194990 in provider escience_a40 proc_lsst.multi INFO: found job 17194990 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:21 proc_lsst.multi:146 [INFO] found job 17194991 in provider escience_a40 proc_lsst.multi INFO: found job 17194991 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:21 proc_lsst.multi:146 [INFO] found job 17194992 in provider escience_a40 proc_lsst.multi INFO: found job 17194992 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:21 proc_lsst.multi:146 [INFO] found job 17194993 in provider escience_a40 proc_lsst.multi INFO: found job 17194993 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:22 proc_lsst.multi:146 [INFO] found job 17194994 in provider escience_a40 proc_lsst.multi INFO: found job 17194994 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:22 proc_lsst.multi:146 [INFO] found job 17194995 in provider escience_a40 proc_lsst.multi INFO: found job 17194995 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:22 proc_lsst.multi:146 [INFO] found job 17194996 in provider escience_a40 proc_lsst.multi INFO: found job 17194996 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.dataflow.dflow INFO: Task 63 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2469ed45c39a2b5eb61b09e4d69a11dd with result from task 63 parsl.dataflow.dflow DEBUG: Task 63 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/d8983150-59e0-4526-8c24-b2d372d4110f_isrForCrosstalkSources_1039909_44.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/d8983150-59e0-4526-8c24-b2d372d4110f_isrForCrosstalkSources_1039909_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:23 proc_lsst.multi:146 [INFO] found job 17194997 in provider escience_a40 proc_lsst.multi INFO: found job 17194997 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:23 proc_lsst.multi:146 [INFO] found job 17194998 in provider escience_a40 proc_lsst.multi INFO: found job 17194998 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.dataflow.dflow INFO: Task 64 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5bb91ed03c73be89202e8f7723a36660 with result from task 64 parsl.dataflow.dflow DEBUG: Task 64 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/83f60e4a-f9b4-4402-869a-ba7629753845_isrForCrosstalkSources_1039907_44.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/83f60e4a-f9b4-4402-869a-ba7629753845_isrForCrosstalkSources_1039907_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:23 proc_lsst.multi:146 [INFO] found job 17194999 in provider escience_a40 proc_lsst.multi INFO: found job 17194999 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:24 proc_lsst.multi:146 [INFO] found job 17195000 in provider escience_a40 proc_lsst.multi INFO: found job 17195000 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:24 proc_lsst.multi:146 [INFO] found job 17195001 in provider escience_a40 proc_lsst.multi INFO: found job 17195001 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.dataflow.dflow INFO: Task 69 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d96c78a45ea0790f34cf7f25ef265fb9 with result from task 69 parsl.dataflow.dflow DEBUG: Task 69 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/b6349407-29c9-416b-88a5-9f0c1ccbfcfd_isrForCrosstalkSources_1039907_26.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/b6349407-29c9-416b-88a5-9f0c1ccbfcfd_isrForCrosstalkSources_1039907_26.stderr parsl.dataflow.dflow INFO: Task 88 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49ad55a65d169ce93b6af9832b5485d6 with result from task 88 parsl.dataflow.dflow DEBUG: Task 88 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/da2738f9-9bdc-40fa-a01c-42e7344aaa96_isrForCrosstalkSources_1039909_36.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/da2738f9-9bdc-40fa-a01c-42e7344aaa96_isrForCrosstalkSources_1039909_36.stderr parsl.dataflow.dflow INFO: Task 65 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec8d2f1dbb911e54d2fdfc1af0db0f60 with result from task 65 parsl.dataflow.dflow DEBUG: Task 65 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/73d7d04f-71e4-4a86-8cd3-d098fab3742c_isrForCrosstalkSources_1040013_48.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/73d7d04f-71e4-4a86-8cd3-d098fab3742c_isrForCrosstalkSources_1040013_48.stderr parsl.dataflow.dflow INFO: Task 84 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 696795cbd889adf752eed24753356c10 with result from task 84 parsl.dataflow.dflow DEBUG: Task 84 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/a69ba276-d84b-427a-a5d3-31279cde6a9c_isrForCrosstalkSources_1039908_26.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/a69ba276-d84b-427a-a5d3-31279cde6a9c_isrForCrosstalkSources_1039908_26.stderr parsl.dataflow.dflow INFO: Task 80 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cee26527dd377d11d2b4c12c8d98df91 with result from task 80 parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:25 proc_lsst.multi:146 [INFO] found job 17195002 in provider escience_a40 proc_lsst.multi INFO: found job 17195002 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.dataflow.dflow DEBUG: Task 80 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/2f0cfba8-9e82-4f6f-9b70-3682224eed0c_isrForCrosstalkSources_1040013_18.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/2f0cfba8-9e82-4f6f-9b70-3682224eed0c_isrForCrosstalkSources_1040013_18.stderr parsl.dataflow.dflow INFO: Task 83 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1fed8401e4e557e97f6e7499c51adc3 with result from task 83 parsl.dataflow.dflow DEBUG: Task 83 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/b7993104-e532-4702-b805-d319cfe03a54_isrForCrosstalkSources_1040013_11.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/b7993104-e532-4702-b805-d319cfe03a54_isrForCrosstalkSources_1040013_11.stderr parsl.dataflow.dflow INFO: Task 79 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8fe11efa30dc98c88575ed7517e78af8 with result from task 79 parsl.dataflow.dflow DEBUG: Task 79 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/5653a213-2721-43e5-9bfe-98a7dec7334f_isrForCrosstalkSources_1039909_5.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/5653a213-2721-43e5-9bfe-98a7dec7334f_isrForCrosstalkSources_1039909_5.stderr parsl.dataflow.dflow INFO: Task 86 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9f1bc3dd9dbd72b26da929ea4de5199 with result from task 86 parsl.dataflow.dflow DEBUG: Task 86 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/a2334934-6daa-496e-90ab-4988cd430900_isrForCrosstalkSources_1040013_10.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/a2334934-6daa-496e-90ab-4988cd430900_isrForCrosstalkSources_1040013_10.stderr parsl.dataflow.dflow INFO: Task 66 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0cbdb6974ac51cdea78e0d744fcc9898 with result from task 66 parsl.dataflow.dflow DEBUG: Task 66 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/2c89fb10-67ee-4007-a5e1-f54cc0cd7238_isrForCrosstalkSources_1039909_53.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/2c89fb10-67ee-4007-a5e1-f54cc0cd7238_isrForCrosstalkSources_1039909_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:25 proc_lsst.multi:146 [INFO] found job 17195003 in provider escience_a40 proc_lsst.multi INFO: found job 17195003 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.dataflow.dflow INFO: Task 87 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54fba683628aae716e26444e3d857180 with result from task 87 parsl.dataflow.dflow DEBUG: Task 87 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/a8335bab-9d41-4777-bead-83717ae172b7_isrForCrosstalkSources_1039909_41.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/a8335bab-9d41-4777-bead-83717ae172b7_isrForCrosstalkSources_1039909_41.stderr parsl.dataflow.dflow INFO: Task 75 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f84f8b7c8c3a552e78ca7edb3efdb3d with result from task 75 parsl.dataflow.dflow DEBUG: Task 75 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/72a58af0-9b9c-4b40-a920-1d674615286c_isrForCrosstalkSources_1039909_38.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/72a58af0-9b9c-4b40-a920-1d674615286c_isrForCrosstalkSources_1039909_38.stderr parsl.dataflow.dflow INFO: Task 82 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9d6e20e9223a9ca6b768c9e2c71f6dd with result from task 82 parsl.dataflow.dflow DEBUG: Task 82 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/f9ac4c6d-d633-4044-b6d1-1570d82ee032_isrForCrosstalkSources_1039909_22.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/f9ac4c6d-d633-4044-b6d1-1570d82ee032_isrForCrosstalkSources_1039909_22.stderr parsl.dataflow.dflow INFO: Task 89 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50f4e72be47ba5bb0ae5073184004c77 with result from task 89 parsl.dataflow.dflow DEBUG: Task 89 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/e59f21aa-7a7f-4b80-a388-cda320736c65_isrForCrosstalkSources_1039909_42.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/e59f21aa-7a7f-4b80-a388-cda320736c65_isrForCrosstalkSources_1039909_42.stderr parsl.dataflow.dflow INFO: Task 81 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1cbae554b0afe1a4cbd9b53751e12409 with result from task 81 parsl.dataflow.dflow DEBUG: Task 81 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/e19c88ff-8f01-40c7-a181-996a7ffd3e20_isrForCrosstalkSources_1040013_60.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/e19c88ff-8f01-40c7-a181-996a7ffd3e20_isrForCrosstalkSources_1040013_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:26 proc_lsst.multi:146 [INFO] found job 17195004 in provider escience_a40 proc_lsst.multi INFO: found job 17195004 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.dataflow.dflow INFO: Task 77 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4aec33b8df9778b8f7d6ad6fe1e0ad55 with result from task 77 parsl.dataflow.dflow DEBUG: Task 77 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/41a170b3-f169-45ef-b591-ffbd73e736fe_isrForCrosstalkSources_1039909_58.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/41a170b3-f169-45ef-b591-ffbd73e736fe_isrForCrosstalkSources_1039909_58.stderr parsl.dataflow.dflow INFO: Task 91 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3311950ade074e9e28d1e9989cceeba2 with result from task 91 parsl.dataflow.dflow DEBUG: Task 91 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/7ba4f7b7-fc8c-4174-a142-ac551ccfc8e3_isrForCrosstalkSources_1040013_51.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/7ba4f7b7-fc8c-4174-a142-ac551ccfc8e3_isrForCrosstalkSources_1040013_51.stderr parsl.dataflow.dflow INFO: Task 78 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a810fc65105d074c801e4baaa1a6c731 with result from task 78 parsl.dataflow.dflow DEBUG: Task 78 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/3d5958e0-f2dc-45ab-9f07-6bc8c0ccab1d_isrForCrosstalkSources_1039908_56.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/3d5958e0-f2dc-45ab-9f07-6bc8c0ccab1d_isrForCrosstalkSources_1039908_56.stderr parsl.dataflow.dflow INFO: Task 90 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38de907fd73d5c7f35e25d60807a6165 with result from task 90 parsl.dataflow.dflow DEBUG: Task 90 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/00f7e4c1-9163-4e83-a426-9664540c7887_isrForCrosstalkSources_1039909_1.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/00f7e4c1-9163-4e83-a426-9664540c7887_isrForCrosstalkSources_1039909_1.stderr parsl.dataflow.dflow INFO: Task 93 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e21c7da226ee51ee4ab83bc72a87a4bb with result from task 93 parsl.dataflow.dflow DEBUG: Task 93 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/1c162879-a26b-47b1-a3fe-d1cd8a899b14_isrForCrosstalkSources_1039909_12.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/1c162879-a26b-47b1-a3fe-d1cd8a899b14_isrForCrosstalkSources_1039909_12.stderr parsl.dataflow.dflow INFO: Task 85 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26c506cb7a988830133d7ee7f3338dbc with result from task 85 parsl.dataflow.dflow DEBUG: Task 85 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/7b33a2af-5cbf-4980-888d-b2cbf59acc79_isrForCrosstalkSources_1039908_41.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/7b33a2af-5cbf-4980-888d-b2cbf59acc79_isrForCrosstalkSources_1039908_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:26 proc_lsst.multi:146 [INFO] found job 17195005 in provider escience_a40 proc_lsst.multi INFO: found job 17195005 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.dataflow.dflow INFO: Task 94 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4bbce07c6b3d98990359cf9a36817243 with result from task 94 parsl.dataflow.dflow DEBUG: Task 94 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/6fdfc4e5-3b69-4980-be3c-dde03b68bc88_isrForCrosstalkSources_1039908_59.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/6fdfc4e5-3b69-4980-be3c-dde03b68bc88_isrForCrosstalkSources_1039908_59.stderr parsl.dataflow.dflow INFO: Task 95 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5040afd76698d1807894b39ec79c049b with result from task 95 parsl.dataflow.dflow DEBUG: Task 95 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/cbfd2566-3c7d-4669-a5b4-742e848fe527_isrForCrosstalkSources_1039908_9.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/cbfd2566-3c7d-4669-a5b4-742e848fe527_isrForCrosstalkSources_1039908_9.stderr parsl.dataflow.dflow INFO: Task 92 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66672665f710b42ef38cdd0fa20a7ca4 with result from task 92 parsl.dataflow.dflow DEBUG: Task 92 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/f963eb80-c7b4-442a-911a-afb013859914_isrForCrosstalkSources_1039908_58.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/f963eb80-c7b4-442a-911a-afb013859914_isrForCrosstalkSources_1039908_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:27 proc_lsst.multi:146 [INFO] found job 17195006 in provider escience_a40 proc_lsst.multi INFO: found job 17195006 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.dataflow.dflow INFO: Task 105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c94034e60b1950a4aa051fc4106caf5 with result from task 105 parsl.dataflow.dflow DEBUG: Task 105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/86f7117f-1ce8-458f-8515-e1b681ab6a24_isrForCrosstalkSources_1039909_20.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/86f7117f-1ce8-458f-8515-e1b681ab6a24_isrForCrosstalkSources_1039909_20.stderr parsl.dataflow.dflow INFO: Task 106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d667ca6d8bcb31fb4b4c6098e5873b9c with result from task 106 parsl.dataflow.dflow DEBUG: Task 106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/5c965ba2-638c-44a8-be95-a1c250974166_isrForCrosstalkSources_1039908_23.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/5c965ba2-638c-44a8-be95-a1c250974166_isrForCrosstalkSources_1039908_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:27 proc_lsst.multi:146 [INFO] found job 17195007 in provider escience_a40 proc_lsst.multi INFO: found job 17195007 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:27 proc_lsst.multi:146 [INFO] found job 17195008 in provider escience_a40 proc_lsst.multi INFO: found job 17195008 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:27 proc_lsst.multi:146 [INFO] found job 17195009 in provider checkpoint proc_lsst.multi INFO: found job 17195009 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195009,17195010,17195011,17195012,17195013,17195014' parsl.providers.slurm.slurm DEBUG: squeue returned 17195009 R 17195010 R 17195011 R 17195012 R 17195013 R 17195014 R parsl.providers.slurm.slurm DEBUG: Updating job 17195009 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195010 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195011 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195012 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195013 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195014 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:27 proc_lsst.multi:146 [INFO] found job 17195010 in provider checkpoint proc_lsst.multi INFO: found job 17195010 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195009,17195010,17195011,17195012,17195013,17195014' parsl.dataflow.dflow INFO: Task 109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c945f9372ca0de45c1e62befa6b64ae with result from task 109 parsl.dataflow.dflow DEBUG: Task 109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/5034af32-c369-49be-9a23-488aa3b8f0c0_isrForCrosstalkSources_1039908_14.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/5034af32-c369-49be-9a23-488aa3b8f0c0_isrForCrosstalkSources_1039908_14.stderr parsl.dataflow.dflow INFO: Task 108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7af50bd2a93da5fb5336142b7ea86435 with result from task 108 parsl.dataflow.dflow DEBUG: Task 108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/1520199e-2ca3-4452-bf76-9a18a5b9c02b_isrForCrosstalkSources_1039907_29.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/1520199e-2ca3-4452-bf76-9a18a5b9c02b_isrForCrosstalkSources_1039907_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195009 R 17195010 R 17195011 R 17195012 R 17195013 R 17195014 R parsl.providers.slurm.slurm DEBUG: Updating job 17195009 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195010 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195011 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195012 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195013 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195014 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:27 proc_lsst.multi:146 [INFO] found job 17195011 in provider checkpoint proc_lsst.multi INFO: found job 17195011 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195009,17195010,17195011,17195012,17195013,17195014' parsl.dataflow.dflow INFO: Task 71 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 683861b84d0714c68847ef4b97d6a61c with result from task 71 parsl.dataflow.dflow DEBUG: Task 71 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/a0a75fb3-078e-4cc3-b00f-a713942db044_isrForCrosstalkSources_1039908_32.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/a0a75fb3-078e-4cc3-b00f-a713942db044_isrForCrosstalkSources_1039908_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195009 R 17195010 R 17195011 R 17195012 R 17195013 R 17195014 R parsl.providers.slurm.slurm DEBUG: Updating job 17195009 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195010 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195011 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195012 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195013 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195014 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:27 proc_lsst.multi:146 [INFO] found job 17195012 in provider checkpoint proc_lsst.multi INFO: found job 17195012 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195009,17195010,17195011,17195012,17195013,17195014' parsl.providers.slurm.slurm DEBUG: squeue returned 17195009 R 17195010 R 17195011 R 17195012 R 17195013 R 17195014 R parsl.providers.slurm.slurm DEBUG: Updating job 17195009 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195010 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195011 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195012 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195013 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195014 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:28 proc_lsst.multi:146 [INFO] found job 17195013 in provider checkpoint proc_lsst.multi INFO: found job 17195013 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195009,17195010,17195011,17195012,17195013,17195014' parsl.dataflow.dflow INFO: Task 73 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6f4b5a0da00e05bbd5e67faac568bac with result from task 73 parsl.dataflow.dflow DEBUG: Task 73 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/3f4f8971-68d6-4867-99a0-6c338ee76d3f_isrForCrosstalkSources_1039908_30.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/3f4f8971-68d6-4867-99a0-6c338ee76d3f_isrForCrosstalkSources_1039908_30.stderr parsl.dataflow.dflow INFO: Task 72 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f929b2c7103cf61a25cd6c6e43416e87 with result from task 72 parsl.dataflow.dflow DEBUG: Task 72 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/96febe9e-3ea8-4c3f-87c3-18c952c03669_isrForCrosstalkSources_1040013_50.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/96febe9e-3ea8-4c3f-87c3-18c952c03669_isrForCrosstalkSources_1040013_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195009 R 17195010 R 17195011 R 17195012 R 17195013 R 17195014 R parsl.providers.slurm.slurm DEBUG: Updating job 17195009 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195010 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195011 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195012 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195013 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195014 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:28 proc_lsst.multi:146 [INFO] found job 17195014 in provider checkpoint proc_lsst.multi INFO: found job 17195014 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195009,17195010,17195011,17195012,17195013,17195014' parsl.providers.slurm.slurm DEBUG: squeue returned 17195009 R 17195010 R 17195011 R 17195012 R 17195013 R 17195014 R parsl.providers.slurm.slurm DEBUG: Updating job 17195009 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195010 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195011 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195012 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195013 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195014 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 153 parsl.dataflow.strategy DEBUG: Executor multi has 153 active tasks, 62/0 running/pending blocks, and 89 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23151910176784 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 153 parsl.dataflow.strategy DEBUG: Executor multi has 153 active tasks, 62/0 running/pending blocks, and 89 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23151910176784 parsl.dataflow.dflow INFO: Task 74 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6fd90375920ef91b3d18ad1f070fd8ba with result from task 74 parsl.dataflow.dflow DEBUG: Task 74 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/1d3b6cc3-db38-4b5c-8749-ace3a0c3a1a1_isrForCrosstalkSources_1039909_6.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/1d3b6cc3-db38-4b5c-8749-ace3a0c3a1a1_isrForCrosstalkSources_1039909_6.stderr parsl.dataflow.dflow INFO: Task 97 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b58005a9a0e7da4fa7cdc156f01aecb with result from task 97 parsl.dataflow.dflow DEBUG: Task 97 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/38e3c569-113c-45f5-98af-bc7b7d179cf5_isrForCrosstalkSources_1040013_27.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/38e3c569-113c-45f5-98af-bc7b7d179cf5_isrForCrosstalkSources_1040013_27.stderr parsl.dataflow.dflow INFO: Task 70 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4fa4e651b7428e3b6d6201a1f46b067a with result from task 70 parsl.dataflow.dflow DEBUG: Task 70 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/0f03bec4-ef2b-48d4-8320-ba0d5c81b3e1_isrForCrosstalkSources_1039909_59.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/0f03bec4-ef2b-48d4-8320-ba0d5c81b3e1_isrForCrosstalkSources_1039909_59.stderr parsl.dataflow.dflow INFO: Task 102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a21b334a7e94069bf0e14a3ce1324b12 with result from task 102 parsl.dataflow.dflow DEBUG: Task 102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/eef751d0-892b-4ac0-938f-ffa507de4079_isrForCrosstalkSources_1039907_21.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/eef751d0-892b-4ac0-938f-ffa507de4079_isrForCrosstalkSources_1039907_21.stderr parsl.dataflow.dflow INFO: Task 96 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f66b9d5fbd008bef90b081f270b16b49 with result from task 96 parsl.dataflow.dflow DEBUG: Task 96 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/9caec26a-995a-4b03-8220-317666de406a_isrForCrosstalkSources_1039907_30.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/9caec26a-995a-4b03-8220-317666de406a_isrForCrosstalkSources_1039907_30.stderr parsl.dataflow.dflow INFO: Task 101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb8cd87783f3400dfb7aa4e0d16d8c34 with result from task 101 parsl.dataflow.dflow DEBUG: Task 101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/70eff367-ab19-443e-bc47-158016bbbaa6_isrForCrosstalkSources_1039908_33.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/70eff367-ab19-443e-bc47-158016bbbaa6_isrForCrosstalkSources_1039908_33.stderr parsl.dataflow.dflow INFO: Task 107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98392f01487f6a3d7ef75f9bb2d7722a with result from task 107 parsl.dataflow.dflow DEBUG: Task 107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/5bd88acd-c0b6-4fc3-bb33-b7c22ef45090_isrForCrosstalkSources_1039909_11.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/5bd88acd-c0b6-4fc3-bb33-b7c22ef45090_isrForCrosstalkSources_1039909_11.stderr parsl.dataflow.dflow INFO: Task 98 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14ebeaa341123c6edac90e43cd26dd36 with result from task 98 parsl.dataflow.dflow DEBUG: Task 98 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/a7cb6420-09fb-407b-aba1-d7393e588ea8_isrForCrosstalkSources_1039908_5.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/a7cb6420-09fb-407b-aba1-d7393e588ea8_isrForCrosstalkSources_1039908_5.stderr parsl.dataflow.dflow INFO: Task 103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ba3bc52b967523d1c0ddfb99075923a with result from task 103 parsl.dataflow.dflow DEBUG: Task 103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/a83c5bc1-0c2f-4b96-9490-7493a94559dc_isrForCrosstalkSources_1039908_29.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/a83c5bc1-0c2f-4b96-9490-7493a94559dc_isrForCrosstalkSources_1039908_29.stderr parsl.dataflow.dflow INFO: Task 99 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3cdebb1f0640e7e667cfe6216df40436 with result from task 99 parsl.dataflow.dflow DEBUG: Task 99 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/bd4df630-f0fa-406a-aa70-92822fe3312f_isrForCrosstalkSources_1039909_21.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/bd4df630-f0fa-406a-aa70-92822fe3312f_isrForCrosstalkSources_1039909_21.stderr parsl.dataflow.dflow INFO: Task 104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e0b28eb3089dcf924b0b91f99e8163b with result from task 104 parsl.dataflow.dflow DEBUG: Task 104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/7e0236ca-4d08-4c85-8ca5-77712ab31070_isrForCrosstalkSources_1039909_3.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/7e0236ca-4d08-4c85-8ca5-77712ab31070_isrForCrosstalkSources_1039909_3.stderr parsl.dataflow.dflow INFO: Task 100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94cd655ec95182868248f0d262670def with result from task 100 parsl.dataflow.dflow DEBUG: Task 100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/eecfe7bc-f685-4a54-9b1a-d9c2cbf72bd4_isrForCrosstalkSources_1039909_40.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/eecfe7bc-f685-4a54-9b1a-d9c2cbf72bd4_isrForCrosstalkSources_1039909_40.stderr parsl.dataflow.dflow INFO: Task 110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fc70f1a4813496c3de157220e2b1e6f with result from task 110 parsl.dataflow.dflow DEBUG: Task 110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/495b4694-0dee-4ca4-971f-af0c51268393_isrForCrosstalkSources_1039908_52.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/495b4694-0dee-4ca4-971f-af0c51268393_isrForCrosstalkSources_1039908_52.stderr parsl.dataflow.dflow INFO: Task 111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d43066392bd7f7b0320a04d3f93cd7b6 with result from task 111 parsl.dataflow.dflow DEBUG: Task 111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/794069d1-239a-41f2-a8bd-53478d58ef18_isrForCrosstalkSources_1039908_3.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/794069d1-239a-41f2-a8bd-53478d58ef18_isrForCrosstalkSources_1039908_3.stderr parsl.dataflow.dflow INFO: Task 112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3683d8f302fb761a292890eb4dba3f9c with result from task 112 parsl.dataflow.dflow DEBUG: Task 112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/e063c6f3-0134-4f50-b5a9-1ac5ab33ce0e_isrForCrosstalkSources_1039908_36.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/e063c6f3-0134-4f50-b5a9-1ac5ab33ce0e_isrForCrosstalkSources_1039908_36.stderr parsl.dataflow.dflow INFO: Task 114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d751aeaa75549feda4f55f90009a01a with result from task 114 parsl.dataflow.dflow DEBUG: Task 114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/efbb0037-c71d-46af-bdfa-61a2f4012b19_isrForCrosstalkSources_1039909_33.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/efbb0037-c71d-46af-bdfa-61a2f4012b19_isrForCrosstalkSources_1039909_33.stderr parsl.dataflow.dflow INFO: Task 113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8368312362f210586bebcddead069a4 with result from task 113 parsl.dataflow.dflow DEBUG: Task 113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/abf832e0-7bbf-4574-ac82-92bcc3a34d01_isrForCrosstalkSources_1039908_35.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/abf832e0-7bbf-4574-ac82-92bcc3a34d01_isrForCrosstalkSources_1039908_35.stderr parsl.dataflow.dflow INFO: Task 116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07e18a85076c7ff5cadca3b591a6480a with result from task 116 parsl.dataflow.dflow DEBUG: Task 116 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/c46f2c59-462d-4e2c-b042-0c62c73b473b_isrForCrosstalkSources_1039907_13.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/c46f2c59-462d-4e2c-b042-0c62c73b473b_isrForCrosstalkSources_1039907_13.stderr parsl.dataflow.dflow INFO: Task 117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb3a9c95deda43c62128bbceb9725729 with result from task 117 parsl.dataflow.dflow DEBUG: Task 117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/5c1f3cfe-d786-4541-ba4e-98e1c6d2aab3_isrForCrosstalkSources_1039909_45.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/5c1f3cfe-d786-4541-ba4e-98e1c6d2aab3_isrForCrosstalkSources_1039909_45.stderr parsl.dataflow.dflow INFO: Task 118 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 587358d22fe8cd82f5636db26db09879 with result from task 118 parsl.dataflow.dflow DEBUG: Task 118 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/952954f7-025c-4bef-9cf6-1e222a944736_isrForCrosstalkSources_1039909_48.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/952954f7-025c-4bef-9cf6-1e222a944736_isrForCrosstalkSources_1039909_48.stderr parsl.dataflow.dflow INFO: Task 119 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a47c57b022f49781e685f6b5020fb3a with result from task 119 parsl.dataflow.dflow DEBUG: Task 119 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/c63671b8-ea23-4324-8fb6-7d83072b0f81_isrForCrosstalkSources_1040013_36.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/c63671b8-ea23-4324-8fb6-7d83072b0f81_isrForCrosstalkSources_1040013_36.stderr parsl.dataflow.dflow INFO: Task 115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ecc161530ccc5eafcaadc18f463917e with result from task 115 parsl.dataflow.dflow DEBUG: Task 115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/29eb304f-c51a-45fb-a29a-25d4fbb0e0ec_isrForCrosstalkSources_1039909_57.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/29eb304f-c51a-45fb-a29a-25d4fbb0e0ec_isrForCrosstalkSources_1039909_57.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 131 parsl.dataflow.strategy DEBUG: Executor multi has 131 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23151910176784 parsl.dataflow.dflow INFO: Task 138 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b68c2e1bd9f038365834eeb32458a3d with result from task 138 parsl.dataflow.dflow DEBUG: Task 138 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/7c21b7f5-ff62-4f47-a15c-69061f4f26aa_isrForCrosstalkSources_1040013_4.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/7c21b7f5-ff62-4f47-a15c-69061f4f26aa_isrForCrosstalkSources_1040013_4.stderr parsl.dataflow.dflow INFO: Task 137 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6108fabeb50d64958eaa932e3f1c74f7 with result from task 137 parsl.dataflow.dflow DEBUG: Task 137 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/9e389aa4-734b-4c65-97bc-91ef7c28cbae_isrForCrosstalkSources_1039909_19.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/9e389aa4-734b-4c65-97bc-91ef7c28cbae_isrForCrosstalkSources_1039909_19.stderr parsl.dataflow.dflow INFO: Task 139 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9bc8c7805c25534c66c2e09c62fe9060 with result from task 139 parsl.dataflow.dflow DEBUG: Task 139 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/f3dcc77f-e2c5-471e-aabc-4d0a9dec93d9_isrForCrosstalkSources_1039908_17.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/f3dcc77f-e2c5-471e-aabc-4d0a9dec93d9_isrForCrosstalkSources_1039908_17.stderr parsl.dataflow.dflow INFO: Task 140 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e257e7bcf1fe0e45823875d27c66f6fb with result from task 140 parsl.dataflow.dflow DEBUG: Task 140 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/47419a5a-9165-4826-b748-90568a7bc3c6_isrForCrosstalkSources_1040013_13.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/47419a5a-9165-4826-b748-90568a7bc3c6_isrForCrosstalkSources_1040013_13.stderr parsl.dataflow.dflow INFO: Task 149 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8dc1b87314d9ef6160ea60dc520e0fec with result from task 149 parsl.dataflow.dflow DEBUG: Task 149 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/c0221f90-40c9-4d11-abf0-44c7da746c5e_isrForCrosstalkSources_1040013_21.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/c0221f90-40c9-4d11-abf0-44c7da746c5e_isrForCrosstalkSources_1040013_21.stderr parsl.dataflow.dflow INFO: Task 153 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 328af9961b73ad2dc195dbc5fe2a689a with result from task 153 parsl.dataflow.dflow DEBUG: Task 153 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/94a20535-999c-4ed1-9f56-2d1f0bbe2678_isrForCrosstalkSources_1039907_1.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/94a20535-999c-4ed1-9f56-2d1f0bbe2678_isrForCrosstalkSources_1039907_1.stderr parsl.dataflow.dflow INFO: Task 142 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a136be14da042d7250e4bdfed96c31b with result from task 142 parsl.dataflow.dflow DEBUG: Task 142 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/8fd8bed7-68f3-45f3-84fa-a801840f674f_isrForCrosstalkSources_1039908_42.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/8fd8bed7-68f3-45f3-84fa-a801840f674f_isrForCrosstalkSources_1039908_42.stderr parsl.dataflow.dflow INFO: Task 151 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb5352c52c222adf74466316421de143 with result from task 151 parsl.dataflow.dflow DEBUG: Task 151 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/174dae44-b40b-4ab7-9f84-a43f73c3db5f_isrForCrosstalkSources_1039907_23.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/174dae44-b40b-4ab7-9f84-a43f73c3db5f_isrForCrosstalkSources_1039907_23.stderr parsl.dataflow.dflow INFO: Task 166 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8761af9af7edaa93d33b9b63be6e406a with result from task 166 parsl.dataflow.dflow DEBUG: Task 166 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/e6fb6c48-99bb-4603-9cc2-41c83e8488e7_isrForCrosstalkSources_1039909_24.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/e6fb6c48-99bb-4603-9cc2-41c83e8488e7_isrForCrosstalkSources_1039909_24.stderr parsl.dataflow.dflow INFO: Task 163 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b882d1e7815661e616e96e2c453507c9 with result from task 163 parsl.dataflow.dflow DEBUG: Task 163 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/27213552-4dc5-40f2-85b3-92d4f47b2b9a_isrForCrosstalkSources_1039909_49.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/27213552-4dc5-40f2-85b3-92d4f47b2b9a_isrForCrosstalkSources_1039909_49.stderr parsl.dataflow.dflow INFO: Task 167 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad17c1541bbaabf17514ddf59bd3a797 with result from task 167 parsl.dataflow.dflow DEBUG: Task 167 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/733a4c35-236d-4296-ac5c-77043d6cbe6e_isrForCrosstalkSources_1039909_25.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/733a4c35-236d-4296-ac5c-77043d6cbe6e_isrForCrosstalkSources_1039909_25.stderr parsl.dataflow.dflow INFO: Task 168 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 253090ee0d0553194a58839cecd99582 with result from task 168 parsl.dataflow.dflow DEBUG: Task 168 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/583940cd-e64f-4823-b0f2-c244e242ac10_isrForCrosstalkSources_1039907_10.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/583940cd-e64f-4823-b0f2-c244e242ac10_isrForCrosstalkSources_1039907_10.stderr parsl.dataflow.dflow INFO: Task 164 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c5520a9fcbcb879830941626c04f2e4 with result from task 164 parsl.dataflow.dflow DEBUG: Task 164 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/e5d854ba-afa2-413d-8d5e-759e7d8b9b13_isrForCrosstalkSources_1040013_52.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/e5d854ba-afa2-413d-8d5e-759e7d8b9b13_isrForCrosstalkSources_1040013_52.stderr parsl.dataflow.dflow INFO: Task 178 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 853fe202149e14164bf4f6e857656371 with result from task 178 parsl.dataflow.dflow DEBUG: Task 178 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/af8a5c59-2a11-4f69-86f7-b414e0ef37e7_isrForCrosstalkSources_1039909_34.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/af8a5c59-2a11-4f69-86f7-b414e0ef37e7_isrForCrosstalkSources_1039909_34.stderr parsl.dataflow.dflow INFO: Task 169 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 415939547ed681c2dbbfe680f70c66ff with result from task 169 parsl.dataflow.dflow DEBUG: Task 169 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/c4318a49-fdac-4561-8b4a-b84ae202ee09_isrForCrosstalkSources_1039907_28.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/c4318a49-fdac-4561-8b4a-b84ae202ee09_isrForCrosstalkSources_1039907_28.stderr parsl.dataflow.dflow INFO: Task 165 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea835f9b9523156fe2572cf404af9bb5 with result from task 165 parsl.dataflow.dflow DEBUG: Task 165 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/5aa323a7-091b-4c55-b866-bc22b4f84079_isrForCrosstalkSources_1040013_59.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/5aa323a7-091b-4c55-b866-bc22b4f84079_isrForCrosstalkSources_1040013_59.stderr parsl.dataflow.dflow INFO: Task 136 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf558d570b1becad3bfce274e178e960 with result from task 136 parsl.dataflow.dflow DEBUG: Task 136 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/d9aeb014-cf0b-4aa5-8715-56feac4189a0_isrForCrosstalkSources_1039908_57.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/d9aeb014-cf0b-4aa5-8715-56feac4189a0_isrForCrosstalkSources_1039908_57.stderr parsl.dataflow.dflow INFO: Task 180 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8bd777e78d0b6efc649752931feddf37 with result from task 180 parsl.dataflow.dflow DEBUG: Task 180 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/00023a66-18f4-45bf-8dc9-565f4fd78d65_isrForCrosstalkSources_1039909_8.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/00023a66-18f4-45bf-8dc9-565f4fd78d65_isrForCrosstalkSources_1039909_8.stderr parsl.dataflow.dflow INFO: Task 146 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2cfa5fc032a539f1066405d76d11774c with result from task 146 parsl.dataflow.dflow DEBUG: Task 146 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/1042bcfb-087f-4c28-8d99-541ed33e97cf_isrForCrosstalkSources_1039907_37.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/1042bcfb-087f-4c28-8d99-541ed33e97cf_isrForCrosstalkSources_1039907_37.stderr parsl.dataflow.dflow INFO: Task 147 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12ea48ea902cfcf81b97d29bac503829 with result from task 147 parsl.dataflow.dflow DEBUG: Task 147 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/69f20d65-7fcf-46bb-bc45-dab41860ad20_isrForCrosstalkSources_1040013_28.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/69f20d65-7fcf-46bb-bc45-dab41860ad20_isrForCrosstalkSources_1040013_28.stderr parsl.dataflow.dflow INFO: Task 179 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b204d3edebc892a9cfeacc97349747f with result from task 179 parsl.dataflow.dflow DEBUG: Task 179 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/515bfa2a-8390-41a5-83f1-07e48ec89820_isrForCrosstalkSources_1040013_22.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/515bfa2a-8390-41a5-83f1-07e48ec89820_isrForCrosstalkSources_1040013_22.stderr parsl.dataflow.dflow INFO: Task 152 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc6f7f159fe5221c25445cb5a600776b with result from task 152 parsl.dataflow.dflow DEBUG: Task 152 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/ef0bc175-a1e7-4491-b98e-4f63cec33a60_isrForCrosstalkSources_1039907_17.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/ef0bc175-a1e7-4491-b98e-4f63cec33a60_isrForCrosstalkSources_1039907_17.stderr parsl.dataflow.dflow INFO: Task 144 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc2a56dfcac6fa03fc99abc4f87513e9 with result from task 144 parsl.dataflow.dflow DEBUG: Task 144 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/2b721a52-2ab1-42f1-87e4-91c9fd3d8378_isrForCrosstalkSources_1039907_53.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/2b721a52-2ab1-42f1-87e4-91c9fd3d8378_isrForCrosstalkSources_1039907_53.stderr parsl.dataflow.dflow INFO: Task 145 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5bc7ef44df381f70c1ba59d4c3aebba with result from task 145 parsl.dataflow.dflow DEBUG: Task 145 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/2a6d8a5b-029c-4a5f-a07f-a9db04eb800d_isrForCrosstalkSources_1040013_32.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/2a6d8a5b-029c-4a5f-a07f-a9db04eb800d_isrForCrosstalkSources_1040013_32.stderr parsl.dataflow.dflow INFO: Task 143 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ab17ff80e942ccb7a9bd0332285c9cc with result from task 143 parsl.dataflow.dflow DEBUG: Task 143 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/83aeaacc-3b57-41ba-a283-6010f377c51a_isrForCrosstalkSources_1039907_55.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/83aeaacc-3b57-41ba-a283-6010f377c51a_isrForCrosstalkSources_1039907_55.stderr parsl.dataflow.dflow INFO: Task 154 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae8d051c871909af1cd9d5b70f72c0d3 with result from task 154 parsl.dataflow.dflow DEBUG: Task 154 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/09b2e0d5-842f-4f2e-a675-55a41317ffb2_isrForCrosstalkSources_1039908_34.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/09b2e0d5-842f-4f2e-a675-55a41317ffb2_isrForCrosstalkSources_1039908_34.stderr parsl.dataflow.dflow INFO: Task 150 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry deb123a1d36267423e841c8cfe28772d with result from task 150 parsl.dataflow.dflow DEBUG: Task 150 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/4bd448c6-e5cc-4c86-a3b8-46bb8d71dcd8_isrForCrosstalkSources_1039907_22.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/4bd448c6-e5cc-4c86-a3b8-46bb8d71dcd8_isrForCrosstalkSources_1039907_22.stderr parsl.dataflow.dflow INFO: Task 148 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da235890b7b23d54dbae43ad8ff337aa with result from task 148 parsl.dataflow.dflow DEBUG: Task 148 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/7659ddf9-17b0-44df-b254-819082ac99ff_isrForCrosstalkSources_1040013_49.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/7659ddf9-17b0-44df-b254-819082ac99ff_isrForCrosstalkSources_1040013_49.stderr parsl.dataflow.dflow INFO: Task 141 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f626599b75cea9803b35eafc9da5a8c with result from task 141 parsl.dataflow.dflow DEBUG: Task 141 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/e7d1b2fd-a600-4b19-8296-1d8590843609_isrForCrosstalkSources_1039908_62.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/e7d1b2fd-a600-4b19-8296-1d8590843609_isrForCrosstalkSources_1039908_62.stderr 2024-03-18 05:18:38 proc_lsst.multi:146 [INFO] found job 53289 in provider local proc_lsst.multi INFO: found job 53289 in provider local 2024-03-18 05:18:38 proc_lsst.multi:146 [INFO] found job 17194953 in provider astro proc_lsst.multi INFO: found job 17194953 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.dataflow.dflow INFO: Task 67 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d03c7d094ff363b436db84c11c3709cf with result from task 67 parsl.dataflow.dflow DEBUG: Task 67 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/c8af4291-1112-4ad3-a502-6765c790f5b2_isrForCrosstalkSources_1039908_24.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/c8af4291-1112-4ad3-a502-6765c790f5b2_isrForCrosstalkSources_1039908_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:38 proc_lsst.multi:146 [INFO] found job 17194954 in provider astro proc_lsst.multi INFO: found job 17194954 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.dataflow.dflow INFO: Task 68 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd4e0b29234d0a2a556858e00a737a2d with result from task 68 parsl.dataflow.dflow DEBUG: Task 68 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/f6471099-c717-4ade-b58e-eccfa20a634e_isrForCrosstalkSources_1039909_4.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/f6471099-c717-4ade-b58e-eccfa20a634e_isrForCrosstalkSources_1039909_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:39 proc_lsst.multi:146 [INFO] found job 17194955 in provider astro proc_lsst.multi INFO: found job 17194955 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.dataflow.dflow INFO: Task 185 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60f12620e79352a7147a83362741da6c with result from task 185 parsl.dataflow.dflow DEBUG: Task 185 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/38564380-f2ab-4239-b33a-3f8fd5a8bd8f_isrForCrosstalkSources_1040013_7.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/38564380-f2ab-4239-b33a-3f8fd5a8bd8f_isrForCrosstalkSources_1040013_7.stderr parsl.dataflow.dflow INFO: Task 195 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2bb8438be922a28488eb6e6f06d6cf4 with result from task 195 parsl.dataflow.dflow DEBUG: Task 195 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/a1781469-d62f-494d-8c84-83c3d9d2b4e5_isrForCrosstalkSources_1040013_38.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/a1781469-d62f-494d-8c84-83c3d9d2b4e5_isrForCrosstalkSources_1040013_38.stderr parsl.dataflow.dflow INFO: Task 192 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71e7c5d6d5782c04785a5138d96af84a with result from task 192 parsl.dataflow.dflow DEBUG: Task 192 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/f791af7a-f105-4b6e-bc4a-4470dff713c1_isrForCrosstalkSources_1040013_53.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/f791af7a-f105-4b6e-bc4a-4470dff713c1_isrForCrosstalkSources_1040013_53.stderr parsl.dataflow.dflow INFO: Task 194 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1e824518b608c2c2f4a561f13e3f3b0 with result from task 194 parsl.dataflow.dflow DEBUG: Task 194 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/93488fd1-409e-4420-a988-ad769c90b84c_isrForCrosstalkSources_1039907_34.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/93488fd1-409e-4420-a988-ad769c90b84c_isrForCrosstalkSources_1039907_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:39 proc_lsst.multi:146 [INFO] found job 17194956 in provider astro proc_lsst.multi INFO: found job 17194956 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.dataflow.dflow INFO: Task 190 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0667b6d7880fe72211efcee050476b72 with result from task 190 parsl.dataflow.dflow DEBUG: Task 190 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/adacc621-147f-4b64-8cec-51016987f022_isrForCrosstalkSources_1039907_56.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/adacc621-147f-4b64-8cec-51016987f022_isrForCrosstalkSources_1039907_56.stderr parsl.dataflow.dflow INFO: Task 197 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d67be791962b32f703b3a4a028fb1d3 with result from task 197 parsl.dataflow.dflow DEBUG: Task 197 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/c476bda1-8f7f-4f4d-a86d-6036bbd32ee0_isrForCrosstalkSources_1040013_17.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/c476bda1-8f7f-4f4d-a86d-6036bbd32ee0_isrForCrosstalkSources_1040013_17.stderr parsl.dataflow.dflow INFO: Task 196 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0116da946b6e8d8b371a6c5b004881d with result from task 196 parsl.dataflow.dflow DEBUG: Task 196 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/dafcac66-bf6e-40db-8841-272363b40a6b_isrForCrosstalkSources_1039908_46.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/dafcac66-bf6e-40db-8841-272363b40a6b_isrForCrosstalkSources_1039908_46.stderr parsl.dataflow.dflow INFO: Task 191 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9965eae20fe2e61a3c2b70e119e7f678 with result from task 191 parsl.dataflow.dflow DEBUG: Task 191 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/c1b4f062-9c1c-4986-9e46-a10e456a310e_isrForCrosstalkSources_1039907_12.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/c1b4f062-9c1c-4986-9e46-a10e456a310e_isrForCrosstalkSources_1039907_12.stderr parsl.dataflow.dflow INFO: Task 184 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b60738f09fd6d0a1f616812fa4959482 with result from task 184 parsl.dataflow.dflow DEBUG: Task 184 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/f5a47ea5-cd46-48d1-808a-8571c8bf269f_isrForCrosstalkSources_1039908_13.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/f5a47ea5-cd46-48d1-808a-8571c8bf269f_isrForCrosstalkSources_1039908_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:40 proc_lsst.multi:146 [INFO] found job 17194957 in provider astro proc_lsst.multi INFO: found job 17194957 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.dataflow.dflow INFO: Task 189 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f0f171820ddee2199a3043cded9d33f5 with result from task 189 parsl.dataflow.dflow DEBUG: Task 189 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/b684ebd1-e415-497a-b756-f23a8cfce56e_isrForCrosstalkSources_1039909_35.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/b684ebd1-e415-497a-b756-f23a8cfce56e_isrForCrosstalkSources_1039909_35.stderr parsl.dataflow.dflow INFO: Task 198 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2cb40268c2249db38119a11617fd4042 with result from task 198 parsl.dataflow.dflow DEBUG: Task 198 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/1e7d4758-448f-4be3-81c7-294d90508c5c_isrForCrosstalkSources_1039909_54.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/1e7d4758-448f-4be3-81c7-294d90508c5c_isrForCrosstalkSources_1039909_54.stderr parsl.dataflow.dflow INFO: Task 199 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8066c6a5ed61eb9210f43cc684c33dd0 with result from task 199 parsl.dataflow.dflow DEBUG: Task 199 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/ceeb8b8f-ca3e-438f-a279-3da290caaad4_isrForCrosstalkSources_1039909_61.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/ceeb8b8f-ca3e-438f-a279-3da290caaad4_isrForCrosstalkSources_1039909_61.stderr parsl.dataflow.dflow INFO: Task 201 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9892f05b378ca7e9bb820995d07ba6e with result from task 201 parsl.dataflow.dflow DEBUG: Task 201 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/a8fba6e9-f7cf-46a7-b8ac-1ce9e7d4a422_isrForCrosstalkSources_1039907_5.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/a8fba6e9-f7cf-46a7-b8ac-1ce9e7d4a422_isrForCrosstalkSources_1039907_5.stderr parsl.dataflow.dflow INFO: Task 202 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27d3c66727188bbdee480393efff4b76 with result from task 202 parsl.dataflow.dflow DEBUG: Task 202 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/6e90343d-1517-4f55-99bf-32c32c21fcc5_isrForCrosstalkSources_1040013_45.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/6e90343d-1517-4f55-99bf-32c32c21fcc5_isrForCrosstalkSources_1040013_45.stderr parsl.dataflow.dflow INFO: Task 187 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe0141d342df54d75239095a75aca082 with result from task 187 parsl.dataflow.dflow DEBUG: Task 187 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/cb27bb6e-4b71-43df-99d8-a9231632fa3f_isrForCrosstalkSources_1039908_60.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/cb27bb6e-4b71-43df-99d8-a9231632fa3f_isrForCrosstalkSources_1039908_60.stderr parsl.dataflow.dflow INFO: Task 188 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48a2f39ebe982b825dfbaf0060b6562d with result from task 188 parsl.dataflow.dflow DEBUG: Task 188 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/4495dfe8-b121-4e89-a5a0-3f2fce30f312_isrForCrosstalkSources_1039907_43.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/4495dfe8-b121-4e89-a5a0-3f2fce30f312_isrForCrosstalkSources_1039907_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:40 proc_lsst.multi:146 [INFO] found job 17194958 in provider astro proc_lsst.multi INFO: found job 17194958 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.dataflow.dflow INFO: Task 200 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee3f90f42fac74d8b77c0b6cdad07dda with result from task 200 parsl.dataflow.dflow DEBUG: Task 200 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/176d2ff4-4871-4e3c-a608-676621539399_isrForCrosstalkSources_1039907_49.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/176d2ff4-4871-4e3c-a608-676621539399_isrForCrosstalkSources_1039907_49.stderr parsl.dataflow.dflow INFO: Task 193 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e161956c7023941adde6284467d9d6bd with result from task 193 parsl.dataflow.dflow DEBUG: Task 193 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/f946b7b1-387b-4834-88c6-f90f5495cd93_isrForCrosstalkSources_1039907_50.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/f946b7b1-387b-4834-88c6-f90f5495cd93_isrForCrosstalkSources_1039907_50.stderr parsl.dataflow.dflow INFO: Task 211 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7dcd2f6fcab19b8789f6afc3f467b68 with result from task 211 parsl.dataflow.dflow DEBUG: Task 211 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/61b5faab-fab0-4494-bab0-b56f2de870e9_isrForCrosstalkSources_1039908_11.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/61b5faab-fab0-4494-bab0-b56f2de870e9_isrForCrosstalkSources_1039908_11.stderr parsl.dataflow.dflow INFO: Task 76 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6cdcdf15d546b01bd606918a0c4b32d with result from task 76 parsl.dataflow.dflow DEBUG: Task 76 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/7c12220d-b4ef-4047-8206-6a72afe05e33_isrForCrosstalkSources_1039909_60.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/7c12220d-b4ef-4047-8206-6a72afe05e33_isrForCrosstalkSources_1039909_60.stderr parsl.dataflow.dflow INFO: Task 183 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 745178a06f90788e35e9d7011602a79f with result from task 183 parsl.dataflow.dflow DEBUG: Task 183 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/687b093a-c930-491d-85e2-d7ee29126260_isrForCrosstalkSources_1040013_55.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/687b093a-c930-491d-85e2-d7ee29126260_isrForCrosstalkSources_1040013_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:40 proc_lsst.multi:146 [INFO] found job 17194959 in provider astro proc_lsst.multi INFO: found job 17194959 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.dataflow.dflow INFO: Task 221 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27b7ded2dfa09288c572d0365a465306 with result from task 221 parsl.dataflow.dflow DEBUG: Task 221 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/b1e82eec-2296-4096-950f-166799d246db_isrForCrosstalkSources_1040013_12.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/b1e82eec-2296-4096-950f-166799d246db_isrForCrosstalkSources_1040013_12.stderr parsl.dataflow.dflow INFO: Task 186 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58deb565d486c52a395ea627e04b9f42 with result from task 186 parsl.dataflow.dflow DEBUG: Task 186 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/c19cd28b-693c-4003-ace6-aa6f3dcfc31e_isrForCrosstalkSources_1039908_45.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/c19cd28b-693c-4003-ace6-aa6f3dcfc31e_isrForCrosstalkSources_1039908_45.stderr parsl.dataflow.dflow INFO: Task 122 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12c61b270f41a1bb95e42b34d662002c with result from task 122 parsl.dataflow.dflow DEBUG: Task 122 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/11704442-474e-4c0d-ab30-c94ee1688071_isrForCrosstalkSources_1039907_39.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/11704442-474e-4c0d-ab30-c94ee1688071_isrForCrosstalkSources_1039907_39.stderr parsl.dataflow.dflow INFO: Task 126 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 165c92c41c95d1d28660ee6656d4ce80 with result from task 126 parsl.dataflow.dflow DEBUG: Task 126 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/d4d3d269-c959-498f-8543-67da86401440_isrForCrosstalkSources_1039909_37.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/d4d3d269-c959-498f-8543-67da86401440_isrForCrosstalkSources_1039909_37.stderr parsl.dataflow.dflow INFO: Task 124 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa86bfff37a3a7cb9c4cab02fa5c2bac with result from task 124 parsl.dataflow.dflow DEBUG: Task 124 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/5077302a-aa8e-4f96-a90b-39da603c8823_isrForCrosstalkSources_1039909_9.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/5077302a-aa8e-4f96-a90b-39da603c8823_isrForCrosstalkSources_1039909_9.stderr parsl.dataflow.dflow INFO: Task 120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 892dfd9e57980bc7f4589aef3866c848 with result from task 120 parsl.dataflow.dflow DEBUG: Task 120 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/545bb221-d937-47df-9f0c-022d868d69cc_isrForCrosstalkSources_1039909_13.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/545bb221-d937-47df-9f0c-022d868d69cc_isrForCrosstalkSources_1039909_13.stderr parsl.dataflow.dflow INFO: Task 125 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44bfda861f6f6075b2dde57e896c8f28 with result from task 125 parsl.dataflow.dflow DEBUG: Task 125 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/6bce05dc-214f-4aee-9a9a-75c1382f1361_isrForCrosstalkSources_1040013_56.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/6bce05dc-214f-4aee-9a9a-75c1382f1361_isrForCrosstalkSources_1040013_56.stderr parsl.dataflow.dflow INFO: Task 121 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5bbcf0053c3d3cd105220dfcd03e34a3 with result from task 121 parsl.dataflow.dflow DEBUG: Task 121 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/7b7dd27f-c630-487e-916d-8114f186eecf_isrForCrosstalkSources_1040013_9.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/7b7dd27f-c630-487e-916d-8114f186eecf_isrForCrosstalkSources_1040013_9.stderr parsl.dataflow.dflow INFO: Task 181 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c405b9f05ab633fb7bb603868c31891 with result from task 181 parsl.dataflow.dflow DEBUG: Task 181 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/70331391-8a80-4e88-b655-fa6965764102_isrForCrosstalkSources_1039907_58.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/70331391-8a80-4e88-b655-fa6965764102_isrForCrosstalkSources_1039907_58.stderr parsl.dataflow.dflow INFO: Task 127 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f31d06ef2b43624e422acc4847c01665 with result from task 127 parsl.dataflow.dflow DEBUG: Task 127 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/90736563-56c0-4754-a3e7-18f0be219029_isrForCrosstalkSources_1039908_22.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/90736563-56c0-4754-a3e7-18f0be219029_isrForCrosstalkSources_1039908_22.stderr parsl.dataflow.dflow INFO: Task 220 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a48d7cf0a48b9ce4700c60d6d8af842f with result from task 220 parsl.dataflow.dflow DEBUG: Task 220 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/6e2fc366-bb71-4de3-993b-328d808b9080_isrForCrosstalkSources_1039909_43.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/6e2fc366-bb71-4de3-993b-328d808b9080_isrForCrosstalkSources_1039909_43.stderr parsl.dataflow.dflow INFO: Task 182 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be83056909a3df64ef21abc14c5f4267 with result from task 182 parsl.dataflow.dflow DEBUG: Task 182 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/ced119e1-d8aa-4017-b1db-1d5203f5e475_isrForCrosstalkSources_1039907_2.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/ced119e1-d8aa-4017-b1db-1d5203f5e475_isrForCrosstalkSources_1039907_2.stderr parsl.dataflow.dflow INFO: Task 133 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8a0f1bf3db8076390c41014dbdc2e53 with result from task 133 parsl.dataflow.dflow DEBUG: Task 133 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/68783319-a47f-4548-9037-e066ba21c89f_isrForCrosstalkSources_1040013_34.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/68783319-a47f-4548-9037-e066ba21c89f_isrForCrosstalkSources_1040013_34.stderr parsl.dataflow.dflow INFO: Task 130 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86a93fa04afbd6f31f4d37ca9d76454f with result from task 130 parsl.dataflow.dflow DEBUG: Task 130 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/7e667c07-9f08-41a6-a055-c166cd33dda1_isrForCrosstalkSources_1040013_16.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/7e667c07-9f08-41a6-a055-c166cd33dda1_isrForCrosstalkSources_1040013_16.stderr parsl.dataflow.dflow INFO: Task 135 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e34a14970260744e6293823057f449c with result from task 135 parsl.dataflow.dflow DEBUG: Task 135 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/0d05105e-d8a8-40ed-a92b-044b78d66992_isrForCrosstalkSources_1040013_58.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/0d05105e-d8a8-40ed-a92b-044b78d66992_isrForCrosstalkSources_1040013_58.stderr parsl.dataflow.dflow INFO: Task 132 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e71629fc47ebad28a8d43ffb5bee6059 with result from task 132 parsl.dataflow.dflow DEBUG: Task 132 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/e6c20bab-6380-4068-b504-c251585cdd7a_isrForCrosstalkSources_1039908_40.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/e6c20bab-6380-4068-b504-c251585cdd7a_isrForCrosstalkSources_1039908_40.stderr parsl.dataflow.dflow INFO: Task 131 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2cde382b89b606bab2d507b0127e2056 with result from task 131 parsl.dataflow.dflow DEBUG: Task 131 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/8581258d-3f2d-49c5-a839-5b39208018aa_isrForCrosstalkSources_1039908_10.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/8581258d-3f2d-49c5-a839-5b39208018aa_isrForCrosstalkSources_1039908_10.stderr parsl.dataflow.dflow INFO: Task 128 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6de41fe2426af97774ebf2458292c711 with result from task 128 parsl.dataflow.dflow DEBUG: Task 128 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/3ca05ebb-b58e-45bb-a877-ec36645b9904_isrForCrosstalkSources_1039909_2.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/3ca05ebb-b58e-45bb-a877-ec36645b9904_isrForCrosstalkSources_1039909_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:41 proc_lsst.multi:146 [INFO] found job 17194960 in provider astro proc_lsst.multi INFO: found job 17194960 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.dataflow.dflow INFO: Task 134 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3459a06a74e8fad6c9190a3ac3e365fb with result from task 134 parsl.dataflow.dflow DEBUG: Task 134 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/f345aa69-44e8-48b3-901c-37413ad3e125_isrForCrosstalkSources_1039908_38.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/f345aa69-44e8-48b3-901c-37413ad3e125_isrForCrosstalkSources_1039908_38.stderr parsl.dataflow.dflow INFO: Task 123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f1887b03e0d5746f078c6faee15d597 with result from task 123 parsl.dataflow.dflow DEBUG: Task 123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/b8f80921-ba6b-45c7-bf69-1d9c3caf2479_isrForCrosstalkSources_1039907_57.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/b8f80921-ba6b-45c7-bf69-1d9c3caf2479_isrForCrosstalkSources_1039907_57.stderr parsl.dataflow.dflow INFO: Task 129 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9923f3f16736f3a4487bf99aee85bef4 with result from task 129 parsl.dataflow.dflow DEBUG: Task 129 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/5e3d4f6b-2683-41b5-9603-bb2a57a90131_isrForCrosstalkSources_1039908_54.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/5e3d4f6b-2683-41b5-9603-bb2a57a90131_isrForCrosstalkSources_1039908_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:42 proc_lsst.multi:146 [INFO] found job 17194961 in provider astro proc_lsst.multi INFO: found job 17194961 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:42 proc_lsst.multi:146 [INFO] found job 17194962 in provider astro proc_lsst.multi INFO: found job 17194962 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:42 proc_lsst.multi:146 [INFO] found job 17194963 in provider astro proc_lsst.multi INFO: found job 17194963 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:43 proc_lsst.multi:146 [INFO] found job 17194964 in provider astro proc_lsst.multi INFO: found job 17194964 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.dataflow.dflow INFO: Task 223 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e19872edc02d69661e4ea3ab80cc2d6 with result from task 223 parsl.dataflow.dflow DEBUG: Task 223 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/51ef72c5-9243-4cf2-8de8-ffe57de96a18_isrForCrosstalkSources_1039908_2.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/51ef72c5-9243-4cf2-8de8-ffe57de96a18_isrForCrosstalkSources_1039908_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:43 proc_lsst.multi:146 [INFO] found job 17194965 in provider astro proc_lsst.multi INFO: found job 17194965 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:43 proc_lsst.multi:146 [INFO] found job 17194966 in provider astro proc_lsst.multi INFO: found job 17194966 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:43 proc_lsst.multi:146 [INFO] found job 17194967 in provider astro proc_lsst.multi INFO: found job 17194967 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:43 proc_lsst.multi:146 [INFO] found job 17194968 in provider astro proc_lsst.multi INFO: found job 17194968 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:43 proc_lsst.multi:146 [INFO] found job 17194969 in provider astro proc_lsst.multi INFO: found job 17194969 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:44 proc_lsst.multi:146 [INFO] found job 17194970 in provider astro proc_lsst.multi INFO: found job 17194970 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:44 proc_lsst.multi:146 [INFO] found job 17194971 in provider astro proc_lsst.multi INFO: found job 17194971 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.dataflow.dflow INFO: Task 227 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0f9c5bff8dd79c6232ba74d1d83ca54 with result from task 227 parsl.dataflow.dflow DEBUG: Task 227 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/9e0fb149-0b41-462f-b97d-83bc0dbbbc27_isrForCrosstalkSources_1040013_61.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/9e0fb149-0b41-462f-b97d-83bc0dbbbc27_isrForCrosstalkSources_1040013_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:44 proc_lsst.multi:146 [INFO] found job 17194973 in provider astro proc_lsst.multi INFO: found job 17194973 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.dataflow.dflow INFO: Task 234 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c96dbd2b4eef7639afbfafb12386469e with result from task 234 parsl.dataflow.dflow DEBUG: Task 234 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/7cfdd951-f0ea-49ec-9ebb-3ac84e73745a_isrForCrosstalkSources_1039909_39.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/7cfdd951-f0ea-49ec-9ebb-3ac84e73745a_isrForCrosstalkSources_1039909_39.stderr parsl.dataflow.dflow INFO: Task 236 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ad36f2b1151192a294e2c5846ae7579 with result from task 236 parsl.dataflow.dflow DEBUG: Task 236 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/6da9929f-00c7-4c0b-8914-5c7ef1569d86_isrForCrosstalkSources_1040013_42.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/6da9929f-00c7-4c0b-8914-5c7ef1569d86_isrForCrosstalkSources_1040013_42.stderr parsl.dataflow.dflow INFO: Task 233 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5def4f6dc05edd5e0f826748efeeceb5 with result from task 233 parsl.dataflow.dflow DEBUG: Task 233 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/0f3d783f-5aa3-4a0a-b691-679636502183_isrForCrosstalkSources_1039909_10.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/0f3d783f-5aa3-4a0a-b691-679636502183_isrForCrosstalkSources_1039909_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:44 proc_lsst.multi:146 [INFO] found job 17194974 in provider astro proc_lsst.multi INFO: found job 17194974 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:44 proc_lsst.multi:146 [INFO] found job 17194975 in provider astro proc_lsst.multi INFO: found job 17194975 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:44 proc_lsst.multi:146 [INFO] found job 17194976 in provider astro proc_lsst.multi INFO: found job 17194976 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.dataflow.dflow INFO: Task 232 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cdfb80d097b5f959d8dac5bacdc7e6d5 with result from task 232 parsl.dataflow.dflow DEBUG: Task 232 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/a14263de-846b-4c95-b680-43886dbb4f05_isrForCrosstalkSources_1039909_62.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/a14263de-846b-4c95-b680-43886dbb4f05_isrForCrosstalkSources_1039909_62.stderr parsl.dataflow.dflow INFO: Task 239 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78e51f3dd60b97882d03e578b0b8c10d with result from task 239 parsl.dataflow.dflow DEBUG: Task 239 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/8e187786-bca1-4cf8-acc2-cc0ae2e7b5ff_isrForCrosstalkSources_1040013_25.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/8e187786-bca1-4cf8-acc2-cc0ae2e7b5ff_isrForCrosstalkSources_1040013_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:44 proc_lsst.multi:146 [INFO] found job 17194977 in provider astro proc_lsst.multi INFO: found job 17194977 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.dataflow.dflow INFO: Task 238 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f3f33fdfe200364ad41d8fc6f992adc with result from task 238 parsl.dataflow.dflow DEBUG: Task 238 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/3b108fd4-e2b7-4617-9a72-afc60bd3771a_isrForCrosstalkSources_1039907_32.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/3b108fd4-e2b7-4617-9a72-afc60bd3771a_isrForCrosstalkSources_1039907_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:45 proc_lsst.multi:146 [INFO] found job 17194978 in provider astro proc_lsst.multi INFO: found job 17194978 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.dataflow.dflow INFO: Task 230 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75ec5da1d7443b599d601c67aa3e05af with result from task 230 parsl.dataflow.dflow DEBUG: Task 230 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/bf47e2ad-e979-42dc-b8d7-bb0ad6ec84bf_isrForCrosstalkSources_1039908_15.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/bf47e2ad-e979-42dc-b8d7-bb0ad6ec84bf_isrForCrosstalkSources_1039908_15.stderr parsl.dataflow.dflow INFO: Task 231 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eff6b786e760366b33ad5e553e642050 with result from task 231 parsl.dataflow.dflow DEBUG: Task 231 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/06596086-e281-4a11-9734-d20ca3d44548_isrForCrosstalkSources_1039908_37.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/06596086-e281-4a11-9734-d20ca3d44548_isrForCrosstalkSources_1039908_37.stderr parsl.dataflow.dflow INFO: Task 242 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56686eb7803d03948520a28cf9937f83 with result from task 242 parsl.dataflow.dflow DEBUG: Task 242 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/a7acc8d4-1a89-4bc0-8c68-8b39e409bdc4_isrForCrosstalkSources_1040013_46.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/a7acc8d4-1a89-4bc0-8c68-8b39e409bdc4_isrForCrosstalkSources_1040013_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:45 proc_lsst.multi:146 [INFO] found job 17194979 in provider astro proc_lsst.multi INFO: found job 17194979 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:45 proc_lsst.multi:146 [INFO] found job 17194980 in provider astro proc_lsst.multi INFO: found job 17194980 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:45 proc_lsst.multi:146 [INFO] found job 17194981 in provider astro proc_lsst.multi INFO: found job 17194981 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.dataflow.dflow INFO: Task 222 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab2b442f11a0ef7ef37c07056da0cb9c with result from task 222 parsl.dataflow.dflow DEBUG: Task 222 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/8a2db6f0-d447-4c25-abf0-3c7c620a4e70_isrForCrosstalkSources_1040013_19.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/8a2db6f0-d447-4c25-abf0-3c7c620a4e70_isrForCrosstalkSources_1040013_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:45 proc_lsst.multi:146 [INFO] found job 17194982 in provider astro proc_lsst.multi INFO: found job 17194982 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.dataflow.dflow INFO: Task 235 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff4fe5e91813067f26616dba8fd90823 with result from task 235 parsl.dataflow.dflow DEBUG: Task 235 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/fe83e70e-9f22-463a-8ed1-429d2d3e69cc_isrForCrosstalkSources_1039907_60.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/fe83e70e-9f22-463a-8ed1-429d2d3e69cc_isrForCrosstalkSources_1039907_60.stderr parsl.dataflow.dflow INFO: Task 224 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93df9dd1480aff469913daa6f35c9993 with result from task 224 parsl.dataflow.dflow DEBUG: Task 224 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/02134890-4c11-475a-88d4-870a79252eed_isrForCrosstalkSources_1039907_27.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/02134890-4c11-475a-88d4-870a79252eed_isrForCrosstalkSources_1039907_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:45 proc_lsst.multi:146 [INFO] found job 17194983 in provider astro proc_lsst.multi INFO: found job 17194983 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.dataflow.dflow INFO: Task 225 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03e5a1b1585f13457e4b1f85b8bb80b4 with result from task 225 parsl.dataflow.dflow DEBUG: Task 225 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/38376fc6-c154-4584-8ca1-b646e7c75991_isrForCrosstalkSources_1040013_40.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/38376fc6-c154-4584-8ca1-b646e7c75991_isrForCrosstalkSources_1040013_40.stderr parsl.dataflow.dflow INFO: Task 240 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8645ff26da51eed82b0659b38e2f92cc with result from task 240 parsl.dataflow.dflow DEBUG: Task 240 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/e9153574-e001-4abf-be18-78af376a5db3_isrForCrosstalkSources_1040013_24.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/e9153574-e001-4abf-be18-78af376a5db3_isrForCrosstalkSources_1040013_24.stderr parsl.dataflow.dflow INFO: Task 246 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87ba494a9faf2972ceda8179dff01531 with result from task 246 parsl.dataflow.dflow DEBUG: Task 246 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/69e2eae2-6728-414a-9058-ac450645e98f_isrForCrosstalkSources_1040013_14.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/69e2eae2-6728-414a-9058-ac450645e98f_isrForCrosstalkSources_1040013_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:45 proc_lsst.multi:146 [INFO] found job 17194984 in provider escience_rtx6k proc_lsst.multi INFO: found job 17194984 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194984,17194985,17194986,17194987,17194988' parsl.providers.slurm.slurm DEBUG: squeue returned 17194984 R 17194985 R 17194986 R 17194987 R 17194988 R parsl.providers.slurm.slurm DEBUG: Updating job 17194984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194988 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:45 proc_lsst.multi:146 [INFO] found job 17194985 in provider escience_rtx6k proc_lsst.multi INFO: found job 17194985 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194984,17194985,17194986,17194987,17194988' parsl.dataflow.dflow INFO: Task 237 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e90da52f1482e33233467c89d58278f with result from task 237 parsl.dataflow.dflow DEBUG: Task 237 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/c6fbf300-66d1-44b5-8124-e3d0ac7aca06_isrForCrosstalkSources_1039908_4.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/c6fbf300-66d1-44b5-8124-e3d0ac7aca06_isrForCrosstalkSources_1039908_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17194984 R 17194985 R 17194986 R 17194987 R 17194988 R parsl.providers.slurm.slurm DEBUG: Updating job 17194984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194988 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:46 proc_lsst.multi:146 [INFO] found job 17194986 in provider escience_rtx6k proc_lsst.multi INFO: found job 17194986 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194984,17194985,17194986,17194987,17194988' parsl.providers.slurm.slurm DEBUG: squeue returned 17194984 R 17194985 R 17194986 R 17194987 R 17194988 R parsl.providers.slurm.slurm DEBUG: Updating job 17194984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194988 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:46 proc_lsst.multi:146 [INFO] found job 17194987 in provider escience_rtx6k proc_lsst.multi INFO: found job 17194987 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194984,17194985,17194986,17194987,17194988' parsl.dataflow.dflow INFO: Task 243 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 849ed5e67a97ffae2385aa78ab039c77 with result from task 243 parsl.dataflow.dflow DEBUG: Task 243 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/f7fef0bf-9378-4cff-a681-b46c870569b9_isrForCrosstalkSources_1039907_14.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/f7fef0bf-9378-4cff-a681-b46c870569b9_isrForCrosstalkSources_1039907_14.stderr parsl.dataflow.dflow INFO: Task 244 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 743cdfb18970e0c18c46960ba2257d38 with result from task 244 parsl.dataflow.dflow DEBUG: Task 244 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/7169de63-70e8-4109-9491-81a30117058d_isrForCrosstalkSources_1040013_37.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/7169de63-70e8-4109-9491-81a30117058d_isrForCrosstalkSources_1040013_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17194984 R 17194985 R 17194986 R 17194987 R 17194988 R parsl.providers.slurm.slurm DEBUG: Updating job 17194984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194988 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:46 proc_lsst.multi:146 [INFO] found job 17194988 in provider escience_rtx6k proc_lsst.multi INFO: found job 17194988 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194984,17194985,17194986,17194987,17194988' parsl.providers.slurm.slurm DEBUG: squeue returned 17194984 R 17194985 R 17194986 R 17194987 R 17194988 R parsl.providers.slurm.slurm DEBUG: Updating job 17194984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194988 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:46 proc_lsst.multi:146 [INFO] found job 17194989 in provider escience_a40 proc_lsst.multi INFO: found job 17194989 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.dataflow.dflow INFO: Task 226 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee22ef8d064cbb60b47d35763e79e410 with result from task 226 parsl.dataflow.dflow DEBUG: Task 226 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/2dc37206-1fae-477f-83cb-0ff6c74a73c0_isrForCrosstalkSources_1040013_54.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/2dc37206-1fae-477f-83cb-0ff6c74a73c0_isrForCrosstalkSources_1040013_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:46 proc_lsst.multi:146 [INFO] found job 17194990 in provider escience_a40 proc_lsst.multi INFO: found job 17194990 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.dataflow.dflow INFO: Task 228 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30e4b737a4b280c74fc3bc9204a4d18e with result from task 228 parsl.dataflow.dflow DEBUG: Task 228 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/9b3051e1-2a6d-421d-af22-9af74216a6b9_isrForCrosstalkSources_1040013_62.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/9b3051e1-2a6d-421d-af22-9af74216a6b9_isrForCrosstalkSources_1040013_62.stderr parsl.dataflow.dflow INFO: Task 241 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 432f6d2c31461c48c2b85e10f20e4efd with result from task 241 parsl.dataflow.dflow DEBUG: Task 241 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/4fa6b11e-3d36-4900-9e49-58490df99fd7_isrForCrosstalkSources_1040013_6.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/4fa6b11e-3d36-4900-9e49-58490df99fd7_isrForCrosstalkSources_1040013_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:46 proc_lsst.multi:146 [INFO] found job 17194991 in provider escience_a40 proc_lsst.multi INFO: found job 17194991 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:46 proc_lsst.multi:146 [INFO] found job 17194992 in provider escience_a40 proc_lsst.multi INFO: found job 17194992 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:47 proc_lsst.multi:146 [INFO] found job 17194993 in provider escience_a40 proc_lsst.multi INFO: found job 17194993 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:47 proc_lsst.multi:146 [INFO] found job 17194994 in provider escience_a40 proc_lsst.multi INFO: found job 17194994 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.dataflow.dflow INFO: Task 229 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7302e357adbefec16e9d93ea4a1909d3 with result from task 229 parsl.dataflow.dflow DEBUG: Task 229 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/3d56e608-4fac-46ab-85b1-605791d34a68_isrForCrosstalkSources_1039907_59.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/3d56e608-4fac-46ab-85b1-605791d34a68_isrForCrosstalkSources_1039907_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:47 proc_lsst.multi:146 [INFO] found job 17194995 in provider escience_a40 proc_lsst.multi INFO: found job 17194995 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.dataflow.dflow INFO: Task 247 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95c88ee61d51d0e8e1e08a541195e2fd with result from task 247 parsl.dataflow.dflow DEBUG: Task 247 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/0a8cccd3-faaa-4e8c-8412-1f9085fe8d99_isrForCrosstalkSources_1039909_28.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/0a8cccd3-faaa-4e8c-8412-1f9085fe8d99_isrForCrosstalkSources_1039909_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:47 proc_lsst.multi:146 [INFO] found job 17194996 in provider escience_a40 proc_lsst.multi INFO: found job 17194996 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.dataflow.dflow INFO: Task 245 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5c77dc7d4be2bd4b9a5b5436a2fbc5f with result from task 245 parsl.dataflow.dflow DEBUG: Task 245 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/781bc0f6-e4d5-4378-bb55-203575008c14_isrForCrosstalkSources_1039907_52.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/781bc0f6-e4d5-4378-bb55-203575008c14_isrForCrosstalkSources_1039907_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:47 proc_lsst.multi:146 [INFO] found job 17194997 in provider escience_a40 proc_lsst.multi INFO: found job 17194997 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:47 proc_lsst.multi:146 [INFO] found job 17194998 in provider escience_a40 proc_lsst.multi INFO: found job 17194998 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:47 proc_lsst.multi:146 [INFO] found job 17194999 in provider escience_a40 proc_lsst.multi INFO: found job 17194999 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:47 proc_lsst.multi:146 [INFO] found job 17195000 in provider escience_a40 proc_lsst.multi INFO: found job 17195000 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:48 proc_lsst.multi:146 [INFO] found job 17195001 in provider escience_a40 proc_lsst.multi INFO: found job 17195001 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:48 proc_lsst.multi:146 [INFO] found job 17195002 in provider escience_a40 proc_lsst.multi INFO: found job 17195002 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:48 proc_lsst.multi:146 [INFO] found job 17195003 in provider escience_a40 proc_lsst.multi INFO: found job 17195003 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:48 proc_lsst.multi:146 [INFO] found job 17195004 in provider escience_a40 proc_lsst.multi INFO: found job 17195004 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:48 proc_lsst.multi:146 [INFO] found job 17195005 in provider escience_a40 proc_lsst.multi INFO: found job 17195005 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:48 proc_lsst.multi:146 [INFO] found job 17195006 in provider escience_a40 proc_lsst.multi INFO: found job 17195006 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:48 proc_lsst.multi:146 [INFO] found job 17195007 in provider escience_a40 proc_lsst.multi INFO: found job 17195007 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:49 proc_lsst.multi:146 [INFO] found job 17195008 in provider escience_a40 proc_lsst.multi INFO: found job 17195008 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:49 proc_lsst.multi:146 [INFO] found job 17195009 in provider checkpoint proc_lsst.multi INFO: found job 17195009 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195009,17195010,17195011,17195012,17195013,17195014' parsl.providers.slurm.slurm DEBUG: squeue returned 17195009 R 17195010 R 17195011 R 17195012 R 17195013 R 17195014 R parsl.providers.slurm.slurm DEBUG: Updating job 17195009 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195010 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195011 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195012 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195013 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195014 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:49 proc_lsst.multi:146 [INFO] found job 17195010 in provider checkpoint proc_lsst.multi INFO: found job 17195010 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195009,17195010,17195011,17195012,17195013,17195014' parsl.providers.slurm.slurm DEBUG: squeue returned 17195009 R 17195010 R 17195011 R 17195012 R 17195013 R 17195014 R parsl.providers.slurm.slurm DEBUG: Updating job 17195009 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195010 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195011 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195012 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195013 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195014 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:49 proc_lsst.multi:146 [INFO] found job 17195011 in provider checkpoint proc_lsst.multi INFO: found job 17195011 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195009,17195010,17195011,17195012,17195013,17195014' parsl.providers.slurm.slurm DEBUG: squeue returned 17195009 R 17195010 R 17195011 R 17195012 R 17195013 R 17195014 R parsl.providers.slurm.slurm DEBUG: Updating job 17195009 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195010 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195011 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195012 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195013 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195014 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:49 proc_lsst.multi:146 [INFO] found job 17195012 in provider checkpoint proc_lsst.multi INFO: found job 17195012 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195009,17195010,17195011,17195012,17195013,17195014' parsl.providers.slurm.slurm DEBUG: squeue returned 17195009 R 17195010 R 17195011 R 17195012 R 17195013 R 17195014 R parsl.providers.slurm.slurm DEBUG: Updating job 17195009 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195010 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195011 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195012 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195013 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195014 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:49 proc_lsst.multi:146 [INFO] found job 17195013 in provider checkpoint proc_lsst.multi INFO: found job 17195013 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195009,17195010,17195011,17195012,17195013,17195014' parsl.providers.slurm.slurm DEBUG: squeue returned 17195009 R 17195010 R 17195011 R 17195012 R 17195013 R 17195014 R parsl.providers.slurm.slurm DEBUG: Updating job 17195009 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195010 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195011 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195012 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195013 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195014 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:18:49 proc_lsst.multi:146 [INFO] found job 17195014 in provider checkpoint proc_lsst.multi INFO: found job 17195014 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195009,17195010,17195011,17195012,17195013,17195014' parsl.providers.slurm.slurm DEBUG: squeue returned 17195009 R 17195010 R 17195011 R 17195012 R 17195013 R 17195014 R parsl.providers.slurm.slurm DEBUG: Updating job 17195009 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195010 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195011 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195012 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195013 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195014 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 32 parsl.dataflow.strategy DEBUG: Executor multi has 32 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23151910176784 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 32 parsl.dataflow.strategy DEBUG: Executor multi has 32 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23151910176784 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 32 parsl.dataflow.strategy DEBUG: Executor multi has 32 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23151910176784 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 32 parsl.dataflow.strategy DEBUG: Executor multi has 32 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23151910176784 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 32 parsl.dataflow.strategy DEBUG: Executor multi has 32 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23151910176784 parsl.dataflow.dflow INFO: Task 156 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0dcacf9523e5b94eb01fa7da351736da with result from task 156 parsl.dataflow.dflow DEBUG: Task 156 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/9b75c933-fc24-4852-a00a-696ace6fcc71_isrForCrosstalkSources_1039907_45.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/9b75c933-fc24-4852-a00a-696ace6fcc71_isrForCrosstalkSources_1039907_45.stderr parsl.dataflow.dflow INFO: Task 158 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4b121205085ba27f826db26d030bfaa with result from task 158 parsl.dataflow.dflow DEBUG: Task 158 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/a43ed675-cc24-435b-814b-1b4ce243a229_isrForCrosstalkSources_1039908_8.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/a43ed675-cc24-435b-814b-1b4ce243a229_isrForCrosstalkSources_1039908_8.stderr parsl.dataflow.dflow INFO: Task 160 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbdfe5f721aa24ee16399920175f4647 with result from task 160 parsl.dataflow.dflow DEBUG: Task 160 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/a1341065-7714-43bf-b14e-ecd2b7b18b0c_isrForCrosstalkSources_1039908_20.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/a1341065-7714-43bf-b14e-ecd2b7b18b0c_isrForCrosstalkSources_1039908_20.stderr parsl.dataflow.dflow INFO: Task 161 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd6e95faa0f3ccfa6b166d97b77e8b1e with result from task 161 parsl.dataflow.dflow DEBUG: Task 161 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/fb5d3102-2fe6-4693-98d3-fe1f9c77fe5b_isrForCrosstalkSources_1040013_23.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/fb5d3102-2fe6-4693-98d3-fe1f9c77fe5b_isrForCrosstalkSources_1040013_23.stderr parsl.dataflow.dflow INFO: Task 214 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d75812df5e4129bde564aa366c208b50 with result from task 214 parsl.dataflow.dflow DEBUG: Task 214 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/797cafeb-27c0-4738-93f9-8939064169d0_isrForCrosstalkSources_1039909_46.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/797cafeb-27c0-4738-93f9-8939064169d0_isrForCrosstalkSources_1039909_46.stderr parsl.dataflow.dflow INFO: Task 215 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eff627bab069dcce7630a48b96ac2d3a with result from task 215 parsl.dataflow.dflow DEBUG: Task 215 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/be24a218-0adb-4b90-88a9-6862c0064700_isrForCrosstalkSources_1040013_20.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/be24a218-0adb-4b90-88a9-6862c0064700_isrForCrosstalkSources_1040013_20.stderr parsl.dataflow.dflow INFO: Task 212 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f1887285f565840cf43e3c531653abd8 with result from task 212 parsl.dataflow.dflow DEBUG: Task 212 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/64cba130-7c45-4e4f-b2f0-a129460a7314_isrForCrosstalkSources_1040013_26.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/64cba130-7c45-4e4f-b2f0-a129460a7314_isrForCrosstalkSources_1040013_26.stderr parsl.dataflow.dflow INFO: Task 218 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1763bfd1b8a410ad801570c69803e8a9 with result from task 218 parsl.dataflow.dflow DEBUG: Task 218 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/2ee61303-9db4-45c8-ac07-9b588878a965_isrForCrosstalkSources_1039907_16.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/2ee61303-9db4-45c8-ac07-9b588878a965_isrForCrosstalkSources_1039907_16.stderr parsl.dataflow.dflow INFO: Task 219 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef4a507eb1bb1067cdf4078dceb53012 with result from task 219 parsl.dataflow.dflow DEBUG: Task 219 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/0fd4bfcc-295f-438d-83bf-11fd70617747_isrForCrosstalkSources_1039907_18.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/0fd4bfcc-295f-438d-83bf-11fd70617747_isrForCrosstalkSources_1039907_18.stderr parsl.dataflow.dflow INFO: Task 159 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c45fdef28c35935ce1f23a802025fb6 with result from task 159 parsl.dataflow.dflow DEBUG: Task 159 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/e2b903b4-d77d-411a-9c67-0fcec3b41623_isrForCrosstalkSources_1039908_7.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/e2b903b4-d77d-411a-9c67-0fcec3b41623_isrForCrosstalkSources_1039908_7.stderr parsl.dataflow.dflow INFO: Task 155 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a386807c6f2c0ed63fac0163ce835c0 with result from task 155 parsl.dataflow.dflow DEBUG: Task 155 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/4291a003-df36-47f0-a027-21084ba66d97_isrForCrosstalkSources_1040013_43.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/4291a003-df36-47f0-a027-21084ba66d97_isrForCrosstalkSources_1040013_43.stderr parsl.dataflow.dflow INFO: Task 162 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b83620ede6e3408c999180a039ec1b7e with result from task 162 parsl.dataflow.dflow DEBUG: Task 162 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/c362dec3-79c0-4fa6-8c24-fed9b120f3d2_isrForCrosstalkSources_1039907_3.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/c362dec3-79c0-4fa6-8c24-fed9b120f3d2_isrForCrosstalkSources_1039907_3.stderr parsl.dataflow.dflow INFO: Task 157 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2389b38a77ddf94fb04e90133e972f59 with result from task 157 parsl.dataflow.dflow DEBUG: Task 157 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/ed4140b5-a7c1-44a8-9d68-cbbc21beb7ae_isrForCrosstalkSources_1039908_49.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/ed4140b5-a7c1-44a8-9d68-cbbc21beb7ae_isrForCrosstalkSources_1039908_49.stderr parsl.dataflow.dflow INFO: Task 176 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a13861b0c2ef30dad2b857208465d7a3 with result from task 176 parsl.dataflow.dflow DEBUG: Task 176 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/095a7e0c-30c0-4828-a904-8baa4136e611_isrForCrosstalkSources_1039907_46.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/095a7e0c-30c0-4828-a904-8baa4136e611_isrForCrosstalkSources_1039907_46.stderr parsl.dataflow.dflow INFO: Task 177 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62c17519d7f49280be2e939ef730baef with result from task 177 parsl.dataflow.dflow DEBUG: Task 177 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/88f5c07e-e994-42bd-ada7-707306a9754f_isrForCrosstalkSources_1039907_40.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/88f5c07e-e994-42bd-ada7-707306a9754f_isrForCrosstalkSources_1039907_40.stderr parsl.dataflow.dflow INFO: Task 217 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6fc49f0469ac45c86345903f8fced470 with result from task 217 parsl.dataflow.dflow DEBUG: Task 217 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/23fd06f2-c423-4979-813e-08ef08ce23df_isrForCrosstalkSources_1039908_6.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/23fd06f2-c423-4979-813e-08ef08ce23df_isrForCrosstalkSources_1039908_6.stderr parsl.dataflow.dflow INFO: Task 175 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6373749e962978c17b8d5273f3ab18ac with result from task 175 parsl.dataflow.dflow DEBUG: Task 175 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/7073fae5-c922-4c02-bf2f-1bad04bd6f24_isrForCrosstalkSources_1039908_25.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/7073fae5-c922-4c02-bf2f-1bad04bd6f24_isrForCrosstalkSources_1039908_25.stderr parsl.dataflow.dflow INFO: Task 171 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cee90d14433280cdb21c848555144b12 with result from task 171 parsl.dataflow.dflow DEBUG: Task 171 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/ebed470b-b24c-43d4-82c0-c325cd66646c_isrForCrosstalkSources_1039909_23.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/ebed470b-b24c-43d4-82c0-c325cd66646c_isrForCrosstalkSources_1039909_23.stderr parsl.dataflow.dflow INFO: Task 170 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a0d32c194c32b776c4771c0bd130b6f with result from task 170 parsl.dataflow.dflow DEBUG: Task 170 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/4964450c-709f-4862-8e6a-3be38b740621_isrForCrosstalkSources_1039907_33.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/4964450c-709f-4862-8e6a-3be38b740621_isrForCrosstalkSources_1039907_33.stderr parsl.dataflow.dflow INFO: Task 174 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c8d0d4e52781f88c34c23f80bd417fe with result from task 174 parsl.dataflow.dflow DEBUG: Task 174 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/8ea56a28-4c53-4beb-acf4-60c6d31d73d4_isrForCrosstalkSources_1039907_9.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/8ea56a28-4c53-4beb-acf4-60c6d31d73d4_isrForCrosstalkSources_1039907_9.stderr parsl.dataflow.dflow INFO: Task 216 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b9c75b60f19de3cbe1f586883295137 with result from task 216 parsl.dataflow.dflow DEBUG: Task 216 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/3c6052c0-783d-4766-b17d-f63d78ae7a6f_isrForCrosstalkSources_1039909_50.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/3c6052c0-783d-4766-b17d-f63d78ae7a6f_isrForCrosstalkSources_1039909_50.stderr parsl.dataflow.dflow INFO: Task 173 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a9782452f3a050e7ba84dc5fb2e6725 with result from task 173 parsl.dataflow.dflow DEBUG: Task 173 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/22c621cc-9377-45e3-9883-df9655ae3e2c_isrForCrosstalkSources_1039907_36.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/22c621cc-9377-45e3-9883-df9655ae3e2c_isrForCrosstalkSources_1039907_36.stderr parsl.dataflow.dflow INFO: Task 213 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b5686a603437339ebc6fc86de075ef5 with result from task 213 parsl.dataflow.dflow DEBUG: Task 213 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/308a970f-6a5f-4cf0-86ad-87f294faf620_isrForCrosstalkSources_1039909_47.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039909/308a970f-6a5f-4cf0-86ad-87f294faf620_isrForCrosstalkSources_1039909_47.stderr parsl.dataflow.dflow INFO: Task 172 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01e34fffb0ae16b63066d1ada6afa523 with result from task 172 parsl.dataflow.dflow DEBUG: Task 172 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/742ef30b-effe-479b-b6bf-67258ff4aa52_isrForCrosstalkSources_1039908_53.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/742ef30b-effe-479b-b6bf-67258ff4aa52_isrForCrosstalkSources_1039908_53.stderr 2024-03-18 05:19:10 proc_lsst.multi:146 [INFO] found job 53289 in provider local proc_lsst.multi INFO: found job 53289 in provider local 2024-03-18 05:19:10 proc_lsst.multi:146 [INFO] found job 17194953 in provider astro proc_lsst.multi INFO: found job 17194953 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:10 proc_lsst.multi:146 [INFO] found job 17194954 in provider astro proc_lsst.multi INFO: found job 17194954 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:10 proc_lsst.multi:146 [INFO] found job 17194955 in provider astro proc_lsst.multi INFO: found job 17194955 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:10 proc_lsst.multi:146 [INFO] found job 17194956 in provider astro proc_lsst.multi INFO: found job 17194956 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:10 proc_lsst.multi:146 [INFO] found job 17194957 in provider astro proc_lsst.multi INFO: found job 17194957 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:10 proc_lsst.multi:146 [INFO] found job 17194958 in provider astro proc_lsst.multi INFO: found job 17194958 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:10 proc_lsst.multi:146 [INFO] found job 17194959 in provider astro proc_lsst.multi INFO: found job 17194959 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:11 proc_lsst.multi:146 [INFO] found job 17194960 in provider astro proc_lsst.multi INFO: found job 17194960 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:11 proc_lsst.multi:146 [INFO] found job 17194961 in provider astro proc_lsst.multi INFO: found job 17194961 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:11 proc_lsst.multi:146 [INFO] found job 17194962 in provider astro proc_lsst.multi INFO: found job 17194962 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:11 proc_lsst.multi:146 [INFO] found job 17194963 in provider astro proc_lsst.multi INFO: found job 17194963 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:11 proc_lsst.multi:146 [INFO] found job 17194964 in provider astro proc_lsst.multi INFO: found job 17194964 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:12 proc_lsst.multi:146 [INFO] found job 17194965 in provider astro proc_lsst.multi INFO: found job 17194965 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.dataflow.dflow INFO: Task 206 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae21a9faeab28a7679a0cdfec881014b with result from task 206 parsl.dataflow.dflow DEBUG: Task 206 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/06622e6d-5953-43e7-8b87-43aab46f7056_isrForCrosstalkSources_1039907_47.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/06622e6d-5953-43e7-8b87-43aab46f7056_isrForCrosstalkSources_1039907_47.stderr parsl.dataflow.dflow INFO: Task 205 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ce3a218038a1dcf08be088a653f2bb1 with result from task 205 parsl.dataflow.dflow DEBUG: Task 205 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/2358a57d-de85-4700-80fa-2a7244c9cd85_isrForCrosstalkSources_1040013_29.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/2358a57d-de85-4700-80fa-2a7244c9cd85_isrForCrosstalkSources_1040013_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:12 proc_lsst.multi:146 [INFO] found job 17194966 in provider astro proc_lsst.multi INFO: found job 17194966 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.dataflow.dflow INFO: Task 204 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09f9b87ca2004a9b333a07dac11b6225 with result from task 204 parsl.dataflow.dflow DEBUG: Task 204 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/aaec970b-f8e3-491b-8b36-15b0cdcf3689_isrForCrosstalkSources_1040013_57.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/aaec970b-f8e3-491b-8b36-15b0cdcf3689_isrForCrosstalkSources_1040013_57.stderr parsl.dataflow.dflow INFO: Task 203 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0195ead259609a1c71b45cbc26559ff0 with result from task 203 parsl.dataflow.dflow DEBUG: Task 203 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/cac493bb-b3c2-4079-955b-a49124118f90_isrForCrosstalkSources_1039907_15.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/cac493bb-b3c2-4079-955b-a49124118f90_isrForCrosstalkSources_1039907_15.stderr parsl.dataflow.dflow INFO: Task 207 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 776315bffce999babb8eea2bb2591d13 with result from task 207 parsl.dataflow.dflow DEBUG: Task 207 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/9ae58a90-9414-42a6-96ba-5fdb0d10503a_isrForCrosstalkSources_1039907_38.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/9ae58a90-9414-42a6-96ba-5fdb0d10503a_isrForCrosstalkSources_1039907_38.stderr parsl.dataflow.dflow INFO: Task 208 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b102b973500804b48ca54ee32fec0745 with result from task 208 parsl.dataflow.dflow DEBUG: Task 208 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/9f6e756e-6e17-461b-b841-132755ac3896_isrForCrosstalkSources_1039908_39.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039908/9f6e756e-6e17-461b-b841-132755ac3896_isrForCrosstalkSources_1039908_39.stderr parsl.dataflow.dflow INFO: Task 209 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f33d399fa99b733891e2aaaf7357190 with result from task 209 parsl.dataflow.dflow DEBUG: Task 209 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/b25222f8-ea97-4c38-bf39-2815156929aa_isrForCrosstalkSources_1040013_3.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1040013/b25222f8-ea97-4c38-bf39-2815156929aa_isrForCrosstalkSources_1040013_3.stderr parsl.dataflow.dflow INFO: Task 210 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb9d9dbaf90608be916b8c61a251485f with result from task 210 parsl.dataflow.dflow INFO: DFK cleanup initiated parsl.dataflow.dflow INFO: Summary of tasks in DFK: parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0 parsl.dataflow.dflow INFO: Tasks in state States.pending: 0 parsl.dataflow.dflow INFO: Tasks in state States.running: 0 parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 248 parsl.dataflow.dflow INFO: Tasks in state States.failed: 0 parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 0 parsl.dataflow.dflow INFO: Tasks in state States.launched: 0 parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0 parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0 parsl.dataflow.dflow INFO: Tasks in state States.joining: 0 parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0 parsl.dataflow.dflow INFO: End of summary parsl.dataflow.dflow DEBUG: Task 210 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/78f457b3-fd49-4231-b080-d1798e5d41f2_isrForCrosstalkSources_1039907_4.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/submit/DEEP/20211003/B1b/science#step0/20240318T051656Z/logs/isrForCrosstalkSources/1039907/78f457b3-fd49-4231-b080-d1798e5d41f2_isrForCrosstalkSources_1039907_4.stderr parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Closing job status poller parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:12 proc_lsst.multi:146 [INFO] found job 17194967 in provider astro proc_lsst.multi INFO: found job 17194967 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:12 proc_lsst.multi:146 [INFO] found job 17194968 in provider astro proc_lsst.multi INFO: found job 17194968 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:12 proc_lsst.multi:146 [INFO] found job 17194969 in provider astro proc_lsst.multi INFO: found job 17194969 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:13 proc_lsst.multi:146 [INFO] found job 17194970 in provider astro proc_lsst.multi INFO: found job 17194970 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:13 proc_lsst.multi:146 [INFO] found job 17194971 in provider astro proc_lsst.multi INFO: found job 17194971 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:13 proc_lsst.multi:146 [INFO] found job 17194973 in provider astro proc_lsst.multi INFO: found job 17194973 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:13 proc_lsst.multi:146 [INFO] found job 17194974 in provider astro proc_lsst.multi INFO: found job 17194974 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:13 proc_lsst.multi:146 [INFO] found job 17194975 in provider astro proc_lsst.multi INFO: found job 17194975 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:13 proc_lsst.multi:146 [INFO] found job 17194976 in provider astro proc_lsst.multi INFO: found job 17194976 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:13 proc_lsst.multi:146 [INFO] found job 17194977 in provider astro proc_lsst.multi INFO: found job 17194977 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:14 proc_lsst.multi:146 [INFO] found job 17194978 in provider astro proc_lsst.multi INFO: found job 17194978 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:14 proc_lsst.multi:146 [INFO] found job 17194979 in provider astro proc_lsst.multi INFO: found job 17194979 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:14 proc_lsst.multi:146 [INFO] found job 17194980 in provider astro proc_lsst.multi INFO: found job 17194980 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:14 proc_lsst.multi:146 [INFO] found job 17194981 in provider astro proc_lsst.multi INFO: found job 17194981 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:14 proc_lsst.multi:146 [INFO] found job 17194982 in provider astro proc_lsst.multi INFO: found job 17194982 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:14 proc_lsst.multi:146 [INFO] found job 17194983 in provider astro proc_lsst.multi INFO: found job 17194983 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194953,17194954,17194955,17194956,17194957,17194958,17194959,17194960,17194961,17194962,17194963,17194964,17194965,17194966,17194967,17194968,17194969,17194970,17194971,17194973,17194974,17194975,17194976,17194977,17194978,17194979,17194980,17194981,17194982,17194983' parsl.providers.slurm.slurm DEBUG: squeue returned 17194974 R 17194975 R 17194976 R 17194977 R 17194978 R 17194979 R 17194980 R 17194981 R 17194982 R 17194983 R 17194973 R 17194963 R 17194964 R 17194965 R 17194966 R 17194967 R 17194968 R 17194969 R 17194970 R 17194971 R 17194957 R 17194958 R 17194959 R 17194960 R 17194961 R 17194962 R 17194953 R 17194954 R 17194955 R 17194956 R parsl.providers.slurm.slurm DEBUG: Updating job 17194974 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194975 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194976 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194977 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194981 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194973 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194963 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194964 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194965 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194966 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194967 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194968 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194969 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194970 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194971 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194957 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194958 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194959 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194960 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194962 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194953 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194955 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194956 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:14 proc_lsst.multi:146 [INFO] found job 17194984 in provider escience_rtx6k proc_lsst.multi INFO: found job 17194984 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194984,17194985,17194986,17194987,17194988' parsl.providers.slurm.slurm DEBUG: squeue returned 17194984 R 17194985 R 17194986 R 17194987 R 17194988 R parsl.providers.slurm.slurm DEBUG: Updating job 17194984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194988 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:15 proc_lsst.multi:146 [INFO] found job 17194985 in provider escience_rtx6k proc_lsst.multi INFO: found job 17194985 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194984,17194985,17194986,17194987,17194988' parsl.providers.slurm.slurm DEBUG: squeue returned 17194984 R 17194985 R 17194986 R 17194987 R 17194988 R parsl.providers.slurm.slurm DEBUG: Updating job 17194984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194988 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:15 proc_lsst.multi:146 [INFO] found job 17194986 in provider escience_rtx6k proc_lsst.multi INFO: found job 17194986 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194984,17194985,17194986,17194987,17194988' parsl.providers.slurm.slurm DEBUG: squeue returned 17194984 R 17194985 R 17194986 R 17194987 R 17194988 R parsl.providers.slurm.slurm DEBUG: Updating job 17194984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194988 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:15 proc_lsst.multi:146 [INFO] found job 17194987 in provider escience_rtx6k proc_lsst.multi INFO: found job 17194987 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194984,17194985,17194986,17194987,17194988' parsl.providers.slurm.slurm DEBUG: squeue returned 17194984 R 17194985 R 17194986 R 17194987 R 17194988 R parsl.providers.slurm.slurm DEBUG: Updating job 17194984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194988 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:15 proc_lsst.multi:146 [INFO] found job 17194988 in provider escience_rtx6k proc_lsst.multi INFO: found job 17194988 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194984,17194985,17194986,17194987,17194988' parsl.providers.slurm.slurm DEBUG: squeue returned 17194984 R 17194985 R 17194986 R 17194987 R 17194988 R parsl.providers.slurm.slurm DEBUG: Updating job 17194984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194988 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:15 proc_lsst.multi:146 [INFO] found job 17194989 in provider escience_a40 proc_lsst.multi INFO: found job 17194989 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:15 proc_lsst.multi:146 [INFO] found job 17194990 in provider escience_a40 proc_lsst.multi INFO: found job 17194990 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:15 proc_lsst.multi:146 [INFO] found job 17194991 in provider escience_a40 proc_lsst.multi INFO: found job 17194991 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:16 proc_lsst.multi:146 [INFO] found job 17194992 in provider escience_a40 proc_lsst.multi INFO: found job 17194992 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:16 proc_lsst.multi:146 [INFO] found job 17194993 in provider escience_a40 proc_lsst.multi INFO: found job 17194993 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:16 proc_lsst.multi:146 [INFO] found job 17194994 in provider escience_a40 proc_lsst.multi INFO: found job 17194994 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:16 proc_lsst.multi:146 [INFO] found job 17194995 in provider escience_a40 proc_lsst.multi INFO: found job 17194995 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:16 proc_lsst.multi:146 [INFO] found job 17194996 in provider escience_a40 proc_lsst.multi INFO: found job 17194996 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:16 proc_lsst.multi:146 [INFO] found job 17194997 in provider escience_a40 proc_lsst.multi INFO: found job 17194997 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:16 proc_lsst.multi:146 [INFO] found job 17194998 in provider escience_a40 proc_lsst.multi INFO: found job 17194998 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:17 proc_lsst.multi:146 [INFO] found job 17194999 in provider escience_a40 proc_lsst.multi INFO: found job 17194999 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:17 proc_lsst.multi:146 [INFO] found job 17195000 in provider escience_a40 proc_lsst.multi INFO: found job 17195000 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:17 proc_lsst.multi:146 [INFO] found job 17195001 in provider escience_a40 proc_lsst.multi INFO: found job 17195001 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:17 proc_lsst.multi:146 [INFO] found job 17195002 in provider escience_a40 proc_lsst.multi INFO: found job 17195002 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:17 proc_lsst.multi:146 [INFO] found job 17195003 in provider escience_a40 proc_lsst.multi INFO: found job 17195003 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:17 proc_lsst.multi:146 [INFO] found job 17195004 in provider escience_a40 proc_lsst.multi INFO: found job 17195004 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:17 proc_lsst.multi:146 [INFO] found job 17195005 in provider escience_a40 proc_lsst.multi INFO: found job 17195005 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:18 proc_lsst.multi:146 [INFO] found job 17195006 in provider escience_a40 proc_lsst.multi INFO: found job 17195006 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:18 proc_lsst.multi:146 [INFO] found job 17195007 in provider escience_a40 proc_lsst.multi INFO: found job 17195007 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:18 proc_lsst.multi:146 [INFO] found job 17195008 in provider escience_a40 proc_lsst.multi INFO: found job 17195008 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17194989,17194990,17194991,17194992,17194993,17194994,17194995,17194996,17194997,17194998,17194999,17195000,17195001,17195002,17195003,17195004,17195005,17195006,17195007,17195008' parsl.providers.slurm.slurm DEBUG: squeue returned 17195007 R 17195008 R 17195002 R 17195003 R 17195004 R 17195005 R 17195006 R 17194992 R 17194993 R 17194994 R 17194995 R 17194996 R 17194997 R 17194998 R 17194999 R 17195000 R 17195001 R 17194989 R 17194990 R 17194991 R parsl.providers.slurm.slurm DEBUG: Updating job 17195007 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195008 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195002 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195003 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195004 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195005 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195006 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194997 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194998 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194999 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195000 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195001 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17194991 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:18 proc_lsst.multi:146 [INFO] found job 17195009 in provider checkpoint proc_lsst.multi INFO: found job 17195009 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195009,17195010,17195011,17195012,17195013,17195014' parsl.providers.slurm.slurm DEBUG: squeue returned 17195009 R 17195010 R 17195011 R 17195012 R 17195013 R 17195014 R parsl.providers.slurm.slurm DEBUG: Updating job 17195009 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195010 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195011 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195012 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195013 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195014 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:18 proc_lsst.multi:146 [INFO] found job 17195010 in provider checkpoint proc_lsst.multi INFO: found job 17195010 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195009,17195010,17195011,17195012,17195013,17195014' parsl.providers.slurm.slurm DEBUG: squeue returned 17195009 R 17195010 R 17195011 R 17195012 R 17195013 R 17195014 R parsl.providers.slurm.slurm DEBUG: Updating job 17195009 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195010 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195011 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195012 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195013 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195014 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:18 proc_lsst.multi:146 [INFO] found job 17195011 in provider checkpoint proc_lsst.multi INFO: found job 17195011 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195009,17195010,17195011,17195012,17195013,17195014' parsl.providers.slurm.slurm DEBUG: squeue returned 17195009 R 17195010 R 17195011 R 17195012 R 17195013 R 17195014 R parsl.providers.slurm.slurm DEBUG: Updating job 17195009 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195010 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195011 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195012 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195013 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195014 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:18 proc_lsst.multi:146 [INFO] found job 17195012 in provider checkpoint proc_lsst.multi INFO: found job 17195012 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195009,17195010,17195011,17195012,17195013,17195014' parsl.providers.slurm.slurm DEBUG: squeue returned 17195009 R 17195010 R 17195011 R 17195012 R 17195013 R 17195014 R parsl.providers.slurm.slurm DEBUG: Updating job 17195009 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195010 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195011 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195012 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195013 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195014 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:19 proc_lsst.multi:146 [INFO] found job 17195013 in provider checkpoint proc_lsst.multi INFO: found job 17195013 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195009,17195010,17195011,17195012,17195013,17195014' parsl.providers.slurm.slurm DEBUG: squeue returned 17195009 R 17195010 R 17195011 R 17195012 R 17195013 R 17195014 R parsl.providers.slurm.slurm DEBUG: Updating job 17195009 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195010 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195011 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195012 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195013 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195014 with slurm status R to parsl state JobState.RUNNING 2024-03-18 05:19:19 proc_lsst.multi:146 [INFO] found job 17195014 in provider checkpoint proc_lsst.multi INFO: found job 17195014 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17195009,17195010,17195011,17195012,17195013,17195014' parsl.providers.slurm.slurm DEBUG: squeue returned 17195009 R 17195010 R 17195011 R 17195012 R 17195013 R 17195014 R parsl.providers.slurm.slurm DEBUG: Updating job 17195009 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195010 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195011 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195012 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195013 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17195014 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 0 parsl.dataflow.strategy DEBUG: Executor multi has 0 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 1: Executor has no active tasks parsl.dataflow.strategy DEBUG: Strategy case 1b: Executor has no active tasks, and more (62) than minimum blocks (0) parsl.dataflow.strategy DEBUG: Starting idle timer for executor. If idle time exceeds 120.0s, blocks will be scaled in parsl.dataflow.strategy DEBUG: Idle time 7.152557373046875e-06s is less than max_idletime 120.0s for executor multi; not scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23151910176784 parsl.dataflow.dflow INFO: Terminated job status poller parsl.dataflow.dflow INFO: Scaling in and shutting down executors parsl.dataflow.dflow INFO: Scaling in executor multi parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=62, block_ids=[] parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 62 blocks parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 85ee1a47c87a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 85ee1a47c87a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: af30e1be5e77 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: af30e1be5e77 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4df847a47ea6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4df847a47ea6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 216b917068c2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 216b917068c2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7250d28b912f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7250d28b912f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 79e1f4ba8813 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 79e1f4ba8813 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bf699a4b6fdf parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bf699a4b6fdf parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b9d50661cedd parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b9d50661cedd parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9918d9854ba6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9918d9854ba6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e3a7f2ac8b3b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e3a7f2ac8b3b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1059d9402559 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1059d9402559 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ae283b11e5e3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ae283b11e5e3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 81dc9ff87e92 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 81dc9ff87e92 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f39f780570cc parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f39f780570cc parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b62c56382840 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b62c56382840 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 08707c302fbb parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 08707c302fbb parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b48b965c1473 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b48b965c1473 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 62ea4e8d362b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 62ea4e8d362b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 67d6f9e06a1e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 67d6f9e06a1e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4f46ceb885fa parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4f46ceb885fa parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7fe8e5fbdc6c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7fe8e5fbdc6c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2a41e9a148f5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2a41e9a148f5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b8fc8688f3b8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b8fc8688f3b8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bf5297ab1b66 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bf5297ab1b66 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 67825e86e11d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 67825e86e11d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7609aed12caf parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7609aed12caf parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fda4bbf341a0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fda4bbf341a0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 244d4d1a647f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 244d4d1a647f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 32ff540de501 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 32ff540de501 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: caf27303a5ee parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: caf27303a5ee parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fee677e3a591 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fee677e3a591 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fe91e3c3a429 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fe91e3c3a429 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f8264bf47f3c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f8264bf47f3c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 79601f5098fc parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 79601f5098fc parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7df4303c2510 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7df4303c2510 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 93a43dda13f7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 93a43dda13f7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d6357f0659b1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d6357f0659b1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 77c74d5b93e5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 77c74d5b93e5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1e70574130b3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1e70574130b3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d3a74cef2c17 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d3a74cef2c17 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c45863c5fcd7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c45863c5fcd7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ce5dcf6b7315 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ce5dcf6b7315 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1986063dc6b7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1986063dc6b7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fc60bebf96e9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fc60bebf96e9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 40df144cb46a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 40df144cb46a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f497d87935b0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f497d87935b0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1c27a6d71064 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1c27a6d71064 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3ec134da1c3e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3ec134da1c3e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c097d204e0d0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c097d204e0d0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1cbfce5586a6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1cbfce5586a6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 69880bbaf6ac parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 69880bbaf6ac parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a81b285a0867 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a81b285a0867 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 456efb4cdda1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 456efb4cdda1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b2de1944ca5a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b2de1944ca5a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e2d91cdb52f2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e2d91cdb52f2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: efc557fffa8f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: efc557fffa8f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: aa3b016cba32 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: aa3b016cba32 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d2f6c8c7c0a1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d2f6c8c7c0a1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dcbbe1f24281 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dcbbe1f24281 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d6d9bdc3eae5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d6d9bdc3eae5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d57b5c89f863 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d57b5c89f863 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 894b13d48b5f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 894b13d48b5f 2024-03-18 05:19:19 proc_lsst.multi:146 [INFO] found job 17195009 in provider checkpoint proc_lsst.multi INFO: found job 17195009 in provider checkpoint 2024-03-18 05:19:19 proc_lsst.multi:201 [INFO] cancelling 17195009 on provider checkpoint proc_lsst.multi INFO: cancelling 17195009 on provider checkpoint 2024-03-18 05:19:19 proc_lsst.multi:146 [INFO] found job 17195012 in provider checkpoint proc_lsst.multi INFO: found job 17195012 in provider checkpoint 2024-03-18 05:19:19 proc_lsst.multi:201 [INFO] cancelling 17195012 on provider checkpoint proc_lsst.multi INFO: cancelling 17195012 on provider checkpoint 2024-03-18 05:19:19 proc_lsst.multi:146 [INFO] found job 17195010 in provider checkpoint proc_lsst.multi INFO: found job 17195010 in provider checkpoint 2024-03-18 05:19:19 proc_lsst.multi:201 [INFO] cancelling 17195010 on provider checkpoint proc_lsst.multi INFO: cancelling 17195010 on provider checkpoint 2024-03-18 05:19:19 proc_lsst.multi:146 [INFO] found job 17195011 in provider checkpoint proc_lsst.multi INFO: found job 17195011 in provider checkpoint 2024-03-18 05:19:19 proc_lsst.multi:201 [INFO] cancelling 17195011 on provider checkpoint proc_lsst.multi INFO: cancelling 17195011 on provider checkpoint 2024-03-18 05:19:19 proc_lsst.multi:146 [INFO] found job 17195002 in provider escience_a40 proc_lsst.multi INFO: found job 17195002 in provider escience_a40 2024-03-18 05:19:19 proc_lsst.multi:201 [INFO] cancelling 17195002 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195002 on provider escience_a40 2024-03-18 05:19:19 proc_lsst.multi:146 [INFO] found job 17195001 in provider escience_a40 proc_lsst.multi INFO: found job 17195001 in provider escience_a40 2024-03-18 05:19:19 proc_lsst.multi:201 [INFO] cancelling 17195001 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195001 on provider escience_a40 2024-03-18 05:19:19 proc_lsst.multi:146 [INFO] found job 17195006 in provider escience_a40 proc_lsst.multi INFO: found job 17195006 in provider escience_a40 2024-03-18 05:19:19 proc_lsst.multi:201 [INFO] cancelling 17195006 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195006 on provider escience_a40 2024-03-18 05:19:20 proc_lsst.multi:146 [INFO] found job 17194996 in provider escience_a40 proc_lsst.multi INFO: found job 17194996 in provider escience_a40 2024-03-18 05:19:20 proc_lsst.multi:201 [INFO] cancelling 17194996 on provider escience_a40 proc_lsst.multi INFO: cancelling 17194996 on provider escience_a40 2024-03-18 05:19:20 proc_lsst.multi:146 [INFO] found job 17195000 in provider escience_a40 proc_lsst.multi INFO: found job 17195000 in provider escience_a40 2024-03-18 05:19:20 proc_lsst.multi:201 [INFO] cancelling 17195000 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195000 on provider escience_a40 2024-03-18 05:19:20 proc_lsst.multi:146 [INFO] found job 17194998 in provider escience_a40 proc_lsst.multi INFO: found job 17194998 in provider escience_a40 2024-03-18 05:19:20 proc_lsst.multi:201 [INFO] cancelling 17194998 on provider escience_a40 proc_lsst.multi INFO: cancelling 17194998 on provider escience_a40 2024-03-18 05:19:20 proc_lsst.multi:146 [INFO] found job 17195004 in provider escience_a40 proc_lsst.multi INFO: found job 17195004 in provider escience_a40 2024-03-18 05:19:20 proc_lsst.multi:201 [INFO] cancelling 17195004 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195004 on provider escience_a40 2024-03-18 05:19:20 proc_lsst.multi:146 [INFO] found job 17195003 in provider escience_a40 proc_lsst.multi INFO: found job 17195003 in provider escience_a40 2024-03-18 05:19:20 proc_lsst.multi:201 [INFO] cancelling 17195003 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195003 on provider escience_a40 2024-03-18 05:19:20 proc_lsst.multi:146 [INFO] found job 17195007 in provider escience_a40 proc_lsst.multi INFO: found job 17195007 in provider escience_a40 2024-03-18 05:19:20 proc_lsst.multi:201 [INFO] cancelling 17195007 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195007 on provider escience_a40 2024-03-18 05:19:20 proc_lsst.multi:146 [INFO] found job 17195005 in provider escience_a40 proc_lsst.multi INFO: found job 17195005 in provider escience_a40 2024-03-18 05:19:20 proc_lsst.multi:201 [INFO] cancelling 17195005 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195005 on provider escience_a40 2024-03-18 05:19:20 proc_lsst.multi:146 [INFO] found job 17194991 in provider escience_a40 proc_lsst.multi INFO: found job 17194991 in provider escience_a40 2024-03-18 05:19:20 proc_lsst.multi:201 [INFO] cancelling 17194991 on provider escience_a40 proc_lsst.multi INFO: cancelling 17194991 on provider escience_a40 2024-03-18 05:19:20 proc_lsst.multi:146 [INFO] found job 17194989 in provider escience_a40 proc_lsst.multi INFO: found job 17194989 in provider escience_a40 2024-03-18 05:19:20 proc_lsst.multi:201 [INFO] cancelling 17194989 on provider escience_a40 proc_lsst.multi INFO: cancelling 17194989 on provider escience_a40 2024-03-18 05:19:20 proc_lsst.multi:146 [INFO] found job 17194992 in provider escience_a40 proc_lsst.multi INFO: found job 17194992 in provider escience_a40 2024-03-18 05:19:20 proc_lsst.multi:201 [INFO] cancelling 17194992 on provider escience_a40 proc_lsst.multi INFO: cancelling 17194992 on provider escience_a40 2024-03-18 05:19:20 proc_lsst.multi:146 [INFO] found job 17194975 in provider astro proc_lsst.multi INFO: found job 17194975 in provider astro 2024-03-18 05:19:20 proc_lsst.multi:201 [INFO] cancelling 17194975 on provider astro proc_lsst.multi INFO: cancelling 17194975 on provider astro 2024-03-18 05:19:20 proc_lsst.multi:146 [INFO] found job 17194993 in provider escience_a40 proc_lsst.multi INFO: found job 17194993 in provider escience_a40 2024-03-18 05:19:20 proc_lsst.multi:201 [INFO] cancelling 17194993 on provider escience_a40 proc_lsst.multi INFO: cancelling 17194993 on provider escience_a40 2024-03-18 05:19:21 proc_lsst.multi:146 [INFO] found job 17194974 in provider astro proc_lsst.multi INFO: found job 17194974 in provider astro 2024-03-18 05:19:21 proc_lsst.multi:201 [INFO] cancelling 17194974 on provider astro proc_lsst.multi INFO: cancelling 17194974 on provider astro 2024-03-18 05:19:21 proc_lsst.multi:146 [INFO] found job 17194979 in provider astro proc_lsst.multi INFO: found job 17194979 in provider astro 2024-03-18 05:19:21 proc_lsst.multi:201 [INFO] cancelling 17194979 on provider astro proc_lsst.multi INFO: cancelling 17194979 on provider astro 2024-03-18 05:19:21 proc_lsst.multi:146 [INFO] found job 17194981 in provider astro proc_lsst.multi INFO: found job 17194981 in provider astro 2024-03-18 05:19:21 proc_lsst.multi:201 [INFO] cancelling 17194981 on provider astro proc_lsst.multi INFO: cancelling 17194981 on provider astro 2024-03-18 05:19:21 proc_lsst.multi:146 [INFO] found job 53289 in provider local proc_lsst.multi INFO: found job 53289 in provider local 2024-03-18 05:19:21 proc_lsst.multi:201 [INFO] cancelling 53289 on provider local proc_lsst.multi INFO: cancelling 53289 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 53289 2024-03-18 05:19:21 proc_lsst.multi:146 [INFO] found job 17194976 in provider astro proc_lsst.multi INFO: found job 17194976 in provider astro 2024-03-18 05:19:21 proc_lsst.multi:201 [INFO] cancelling 17194976 on provider astro proc_lsst.multi INFO: cancelling 17194976 on provider astro 2024-03-18 05:19:21 proc_lsst.multi:146 [INFO] found job 17194982 in provider astro proc_lsst.multi INFO: found job 17194982 in provider astro 2024-03-18 05:19:21 proc_lsst.multi:201 [INFO] cancelling 17194982 on provider astro proc_lsst.multi INFO: cancelling 17194982 on provider astro 2024-03-18 05:19:21 proc_lsst.multi:146 [INFO] found job 17194978 in provider astro proc_lsst.multi INFO: found job 17194978 in provider astro 2024-03-18 05:19:21 proc_lsst.multi:201 [INFO] cancelling 17194978 on provider astro proc_lsst.multi INFO: cancelling 17194978 on provider astro 2024-03-18 05:19:21 proc_lsst.multi:146 [INFO] found job 17194980 in provider astro proc_lsst.multi INFO: found job 17194980 in provider astro 2024-03-18 05:19:21 proc_lsst.multi:201 [INFO] cancelling 17194980 on provider astro proc_lsst.multi INFO: cancelling 17194980 on provider astro 2024-03-18 05:19:21 proc_lsst.multi:146 [INFO] found job 17194977 in provider astro proc_lsst.multi INFO: found job 17194977 in provider astro 2024-03-18 05:19:21 proc_lsst.multi:201 [INFO] cancelling 17194977 on provider astro proc_lsst.multi INFO: cancelling 17194977 on provider astro 2024-03-18 05:19:21 proc_lsst.multi:146 [INFO] found job 17194983 in provider astro proc_lsst.multi INFO: found job 17194983 in provider astro 2024-03-18 05:19:21 proc_lsst.multi:201 [INFO] cancelling 17194983 on provider astro proc_lsst.multi INFO: cancelling 17194983 on provider astro 2024-03-18 05:19:21 proc_lsst.multi:146 [INFO] found job 17195013 in provider checkpoint proc_lsst.multi INFO: found job 17195013 in provider checkpoint 2024-03-18 05:19:21 proc_lsst.multi:201 [INFO] cancelling 17195013 on provider checkpoint proc_lsst.multi INFO: cancelling 17195013 on provider checkpoint 2024-03-18 05:19:22 proc_lsst.multi:146 [INFO] found job 17195014 in provider checkpoint proc_lsst.multi INFO: found job 17195014 in provider checkpoint 2024-03-18 05:19:22 proc_lsst.multi:201 [INFO] cancelling 17195014 on provider checkpoint proc_lsst.multi INFO: cancelling 17195014 on provider checkpoint 2024-03-18 05:19:22 proc_lsst.multi:146 [INFO] found job 17194984 in provider escience_rtx6k proc_lsst.multi INFO: found job 17194984 in provider escience_rtx6k 2024-03-18 05:19:22 proc_lsst.multi:201 [INFO] cancelling 17194984 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17194984 on provider escience_rtx6k 2024-03-18 05:19:22 proc_lsst.multi:146 [INFO] found job 17194973 in provider astro proc_lsst.multi INFO: found job 17194973 in provider astro 2024-03-18 05:19:22 proc_lsst.multi:201 [INFO] cancelling 17194973 on provider astro proc_lsst.multi INFO: cancelling 17194973 on provider astro 2024-03-18 05:19:22 proc_lsst.multi:146 [INFO] found job 17194986 in provider escience_rtx6k proc_lsst.multi INFO: found job 17194986 in provider escience_rtx6k 2024-03-18 05:19:22 proc_lsst.multi:201 [INFO] cancelling 17194986 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17194986 on provider escience_rtx6k 2024-03-18 05:19:22 proc_lsst.multi:146 [INFO] found job 17194987 in provider escience_rtx6k proc_lsst.multi INFO: found job 17194987 in provider escience_rtx6k 2024-03-18 05:19:22 proc_lsst.multi:201 [INFO] cancelling 17194987 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17194987 on provider escience_rtx6k 2024-03-18 05:19:22 proc_lsst.multi:146 [INFO] found job 17194958 in provider astro proc_lsst.multi INFO: found job 17194958 in provider astro 2024-03-18 05:19:22 proc_lsst.multi:201 [INFO] cancelling 17194958 on provider astro proc_lsst.multi INFO: cancelling 17194958 on provider astro 2024-03-18 05:19:22 proc_lsst.multi:146 [INFO] found job 17194988 in provider escience_rtx6k proc_lsst.multi INFO: found job 17194988 in provider escience_rtx6k 2024-03-18 05:19:22 proc_lsst.multi:201 [INFO] cancelling 17194988 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17194988 on provider escience_rtx6k 2024-03-18 05:19:22 proc_lsst.multi:146 [INFO] found job 17195008 in provider escience_a40 proc_lsst.multi INFO: found job 17195008 in provider escience_a40 2024-03-18 05:19:22 proc_lsst.multi:201 [INFO] cancelling 17195008 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195008 on provider escience_a40 2024-03-18 05:19:22 proc_lsst.multi:146 [INFO] found job 17194962 in provider astro proc_lsst.multi INFO: found job 17194962 in provider astro 2024-03-18 05:19:22 proc_lsst.multi:201 [INFO] cancelling 17194962 on provider astro proc_lsst.multi INFO: cancelling 17194962 on provider astro 2024-03-18 05:19:22 proc_lsst.multi:146 [INFO] found job 17194961 in provider astro proc_lsst.multi INFO: found job 17194961 in provider astro 2024-03-18 05:19:22 proc_lsst.multi:201 [INFO] cancelling 17194961 on provider astro proc_lsst.multi INFO: cancelling 17194961 on provider astro 2024-03-18 05:19:22 proc_lsst.multi:146 [INFO] found job 17194953 in provider astro proc_lsst.multi INFO: found job 17194953 in provider astro 2024-03-18 05:19:22 proc_lsst.multi:201 [INFO] cancelling 17194953 on provider astro proc_lsst.multi INFO: cancelling 17194953 on provider astro 2024-03-18 05:19:23 proc_lsst.multi:146 [INFO] found job 17194967 in provider astro proc_lsst.multi INFO: found job 17194967 in provider astro 2024-03-18 05:19:23 proc_lsst.multi:201 [INFO] cancelling 17194967 on provider astro proc_lsst.multi INFO: cancelling 17194967 on provider astro 2024-03-18 05:19:23 proc_lsst.multi:146 [INFO] found job 17194970 in provider astro proc_lsst.multi INFO: found job 17194970 in provider astro 2024-03-18 05:19:23 proc_lsst.multi:201 [INFO] cancelling 17194970 on provider astro proc_lsst.multi INFO: cancelling 17194970 on provider astro 2024-03-18 05:19:23 proc_lsst.multi:146 [INFO] found job 17194960 in provider astro proc_lsst.multi INFO: found job 17194960 in provider astro 2024-03-18 05:19:23 proc_lsst.multi:201 [INFO] cancelling 17194960 on provider astro proc_lsst.multi INFO: cancelling 17194960 on provider astro 2024-03-18 05:19:23 proc_lsst.multi:146 [INFO] found job 17194966 in provider astro proc_lsst.multi INFO: found job 17194966 in provider astro 2024-03-18 05:19:23 proc_lsst.multi:201 [INFO] cancelling 17194966 on provider astro proc_lsst.multi INFO: cancelling 17194966 on provider astro 2024-03-18 05:19:23 proc_lsst.multi:146 [INFO] found job 17194969 in provider astro proc_lsst.multi INFO: found job 17194969 in provider astro 2024-03-18 05:19:23 proc_lsst.multi:201 [INFO] cancelling 17194969 on provider astro proc_lsst.multi INFO: cancelling 17194969 on provider astro 2024-03-18 05:19:23 proc_lsst.multi:146 [INFO] found job 17194957 in provider astro proc_lsst.multi INFO: found job 17194957 in provider astro 2024-03-18 05:19:23 proc_lsst.multi:201 [INFO] cancelling 17194957 on provider astro proc_lsst.multi INFO: cancelling 17194957 on provider astro 2024-03-18 05:19:23 proc_lsst.multi:146 [INFO] found job 17194963 in provider astro proc_lsst.multi INFO: found job 17194963 in provider astro 2024-03-18 05:19:23 proc_lsst.multi:201 [INFO] cancelling 17194963 on provider astro proc_lsst.multi INFO: cancelling 17194963 on provider astro 2024-03-18 05:19:23 proc_lsst.multi:146 [INFO] found job 17194985 in provider escience_rtx6k proc_lsst.multi INFO: found job 17194985 in provider escience_rtx6k 2024-03-18 05:19:23 proc_lsst.multi:201 [INFO] cancelling 17194985 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17194985 on provider escience_rtx6k 2024-03-18 05:19:23 proc_lsst.multi:146 [INFO] found job 17194955 in provider astro proc_lsst.multi INFO: found job 17194955 in provider astro 2024-03-18 05:19:23 proc_lsst.multi:201 [INFO] cancelling 17194955 on provider astro proc_lsst.multi INFO: cancelling 17194955 on provider astro 2024-03-18 05:19:24 proc_lsst.multi:146 [INFO] found job 17194959 in provider astro proc_lsst.multi INFO: found job 17194959 in provider astro 2024-03-18 05:19:24 proc_lsst.multi:201 [INFO] cancelling 17194959 on provider astro proc_lsst.multi INFO: cancelling 17194959 on provider astro 2024-03-18 05:19:24 proc_lsst.multi:146 [INFO] found job 17194954 in provider astro proc_lsst.multi INFO: found job 17194954 in provider astro 2024-03-18 05:19:24 proc_lsst.multi:201 [INFO] cancelling 17194954 on provider astro proc_lsst.multi INFO: cancelling 17194954 on provider astro 2024-03-18 05:19:24 proc_lsst.multi:146 [INFO] found job 17194965 in provider astro proc_lsst.multi INFO: found job 17194965 in provider astro 2024-03-18 05:19:24 proc_lsst.multi:201 [INFO] cancelling 17194965 on provider astro proc_lsst.multi INFO: cancelling 17194965 on provider astro 2024-03-18 05:19:24 proc_lsst.multi:146 [INFO] found job 17194956 in provider astro proc_lsst.multi INFO: found job 17194956 in provider astro 2024-03-18 05:19:24 proc_lsst.multi:201 [INFO] cancelling 17194956 on provider astro proc_lsst.multi INFO: cancelling 17194956 on provider astro 2024-03-18 05:19:24 proc_lsst.multi:146 [INFO] found job 17194968 in provider astro proc_lsst.multi INFO: found job 17194968 in provider astro 2024-03-18 05:19:24 proc_lsst.multi:201 [INFO] cancelling 17194968 on provider astro proc_lsst.multi INFO: cancelling 17194968 on provider astro 2024-03-18 05:19:24 proc_lsst.multi:146 [INFO] found job 17194971 in provider astro proc_lsst.multi INFO: found job 17194971 in provider astro 2024-03-18 05:19:24 proc_lsst.multi:201 [INFO] cancelling 17194971 on provider astro proc_lsst.multi INFO: cancelling 17194971 on provider astro 2024-03-18 05:19:24 proc_lsst.multi:146 [INFO] found job 17194964 in provider astro proc_lsst.multi INFO: found job 17194964 in provider astro 2024-03-18 05:19:24 proc_lsst.multi:201 [INFO] cancelling 17194964 on provider astro proc_lsst.multi INFO: cancelling 17194964 on provider astro 2024-03-18 05:19:25 proc_lsst.multi:146 [INFO] found job 17194990 in provider escience_a40 proc_lsst.multi INFO: found job 17194990 in provider escience_a40 2024-03-18 05:19:25 proc_lsst.multi:201 [INFO] cancelling 17194990 on provider escience_a40 proc_lsst.multi INFO: cancelling 17194990 on provider escience_a40 2024-03-18 05:19:25 proc_lsst.multi:146 [INFO] found job 17194994 in provider escience_a40 proc_lsst.multi INFO: found job 17194994 in provider escience_a40 2024-03-18 05:19:25 proc_lsst.multi:201 [INFO] cancelling 17194994 on provider escience_a40 proc_lsst.multi INFO: cancelling 17194994 on provider escience_a40 2024-03-18 05:19:25 proc_lsst.multi:146 [INFO] found job 17194999 in provider escience_a40 proc_lsst.multi INFO: found job 17194999 in provider escience_a40 2024-03-18 05:19:25 proc_lsst.multi:201 [INFO] cancelling 17194999 on provider escience_a40 proc_lsst.multi INFO: cancelling 17194999 on provider escience_a40 2024-03-18 05:19:25 proc_lsst.multi:146 [INFO] found job 17194997 in provider escience_a40 proc_lsst.multi INFO: found job 17194997 in provider escience_a40 2024-03-18 05:19:25 proc_lsst.multi:201 [INFO] cancelling 17194997 on provider escience_a40 proc_lsst.multi INFO: cancelling 17194997 on provider escience_a40 2024-03-18 05:19:25 proc_lsst.multi:146 [INFO] found job 17194995 in provider escience_a40 proc_lsst.multi INFO: found job 17194995 in provider escience_a40 2024-03-18 05:19:25 proc_lsst.multi:201 [INFO] cancelling 17194995 on provider escience_a40 proc_lsst.multi INFO: cancelling 17194995 on provider escience_a40 parsl.dataflow.dflow INFO: Shutting down executor multi 2024-03-18 05:19:25 proc_lsst.multi:40 [INFO] Cancelling all provider resources proc_lsst.multi INFO: Cancelling all provider resources 2024-03-18 05:19:25 proc_lsst.multi:47 [INFO] new jobs since last cancel ['17195009', '17195010', '17195011', '17195012', '17195013', '17195014', '17194953', '17194954', '17194955', '17194956', '17194957', '17194958', '17194959', '17194960', '17194961', '17194962', '17194963', '17194964', '17194965', '17194966', '17194967', '17194968', '17194969', '17194970', '17194971', '17194973', '17194974', '17194975', '17194976', '17194977', '17194978', '17194979', '17194980', '17194981', '17194982', '17194983', '17194989', '17194990', '17194991', '17194992', '17194993', '17194994', '17194995', '17194996', '17194997', '17194998', '17194999', '17195000', '17195001', '17195002', '17195003', '17195004', '17195005', '17195006', '17195007', '17195008', '17194984', '17194985', '17194986', '17194987', '17194988', '53289'] proc_lsst.multi INFO: new jobs since last cancel ['17195009', '17195010', '17195011', '17195012', '17195013', '17195014', '17194953', '17194954', '17194955', '17194956', '17194957', '17194958', '17194959', '17194960', '17194961', '17194962', '17194963', '17194964', '17194965', '17194966', '17194967', '17194968', '17194969', '17194970', '17194971', '17194973', '17194974', '17194975', '17194976', '17194977', '17194978', '17194979', '17194980', '17194981', '17194982', '17194983', '17194989', '17194990', '17194991', '17194992', '17194993', '17194994', '17194995', '17194996', '17194997', '17194998', '17194999', '17195000', '17195001', '17195002', '17195003', '17195004', '17195005', '17195006', '17195007', '17195008', '17194984', '17194985', '17194986', '17194987', '17194988', '53289'] 2024-03-18 05:19:25 proc_lsst.multi:146 [INFO] found job 17195009 in provider checkpoint proc_lsst.multi INFO: found job 17195009 in provider checkpoint 2024-03-18 05:19:25 proc_lsst.multi:201 [INFO] cancelling 17195009 on provider checkpoint proc_lsst.multi INFO: cancelling 17195009 on provider checkpoint 2024-03-18 05:19:25 proc_lsst.multi:146 [INFO] found job 17195010 in provider checkpoint proc_lsst.multi INFO: found job 17195010 in provider checkpoint 2024-03-18 05:19:25 proc_lsst.multi:201 [INFO] cancelling 17195010 on provider checkpoint proc_lsst.multi INFO: cancelling 17195010 on provider checkpoint 2024-03-18 05:19:25 proc_lsst.multi:146 [INFO] found job 17195011 in provider checkpoint proc_lsst.multi INFO: found job 17195011 in provider checkpoint 2024-03-18 05:19:25 proc_lsst.multi:201 [INFO] cancelling 17195011 on provider checkpoint proc_lsst.multi INFO: cancelling 17195011 on provider checkpoint 2024-03-18 05:19:25 proc_lsst.multi:146 [INFO] found job 17195012 in provider checkpoint proc_lsst.multi INFO: found job 17195012 in provider checkpoint 2024-03-18 05:19:25 proc_lsst.multi:201 [INFO] cancelling 17195012 on provider checkpoint proc_lsst.multi INFO: cancelling 17195012 on provider checkpoint 2024-03-18 05:19:25 proc_lsst.multi:146 [INFO] found job 17195013 in provider checkpoint proc_lsst.multi INFO: found job 17195013 in provider checkpoint 2024-03-18 05:19:25 proc_lsst.multi:201 [INFO] cancelling 17195013 on provider checkpoint proc_lsst.multi INFO: cancelling 17195013 on provider checkpoint 2024-03-18 05:19:26 proc_lsst.multi:146 [INFO] found job 17195014 in provider checkpoint proc_lsst.multi INFO: found job 17195014 in provider checkpoint 2024-03-18 05:19:26 proc_lsst.multi:201 [INFO] cancelling 17195014 on provider checkpoint proc_lsst.multi INFO: cancelling 17195014 on provider checkpoint 2024-03-18 05:19:26 proc_lsst.multi:146 [INFO] found job 17194953 in provider astro proc_lsst.multi INFO: found job 17194953 in provider astro 2024-03-18 05:19:26 proc_lsst.multi:201 [INFO] cancelling 17194953 on provider astro proc_lsst.multi INFO: cancelling 17194953 on provider astro 2024-03-18 05:19:26 proc_lsst.multi:146 [INFO] found job 17194954 in provider astro proc_lsst.multi INFO: found job 17194954 in provider astro 2024-03-18 05:19:26 proc_lsst.multi:201 [INFO] cancelling 17194954 on provider astro proc_lsst.multi INFO: cancelling 17194954 on provider astro 2024-03-18 05:19:26 proc_lsst.multi:146 [INFO] found job 17194955 in provider astro proc_lsst.multi INFO: found job 17194955 in provider astro 2024-03-18 05:19:26 proc_lsst.multi:201 [INFO] cancelling 17194955 on provider astro proc_lsst.multi INFO: cancelling 17194955 on provider astro 2024-03-18 05:19:26 proc_lsst.multi:146 [INFO] found job 17194956 in provider astro proc_lsst.multi INFO: found job 17194956 in provider astro 2024-03-18 05:19:26 proc_lsst.multi:201 [INFO] cancelling 17194956 on provider astro proc_lsst.multi INFO: cancelling 17194956 on provider astro 2024-03-18 05:19:26 proc_lsst.multi:146 [INFO] found job 17194957 in provider astro proc_lsst.multi INFO: found job 17194957 in provider astro 2024-03-18 05:19:26 proc_lsst.multi:201 [INFO] cancelling 17194957 on provider astro proc_lsst.multi INFO: cancelling 17194957 on provider astro 2024-03-18 05:19:26 proc_lsst.multi:146 [INFO] found job 17194958 in provider astro proc_lsst.multi INFO: found job 17194958 in provider astro 2024-03-18 05:19:26 proc_lsst.multi:201 [INFO] cancelling 17194958 on provider astro proc_lsst.multi INFO: cancelling 17194958 on provider astro 2024-03-18 05:19:26 proc_lsst.multi:146 [INFO] found job 17194959 in provider astro proc_lsst.multi INFO: found job 17194959 in provider astro 2024-03-18 05:19:26 proc_lsst.multi:201 [INFO] cancelling 17194959 on provider astro proc_lsst.multi INFO: cancelling 17194959 on provider astro 2024-03-18 05:19:26 proc_lsst.multi:146 [INFO] found job 17194960 in provider astro proc_lsst.multi INFO: found job 17194960 in provider astro 2024-03-18 05:19:26 proc_lsst.multi:201 [INFO] cancelling 17194960 on provider astro proc_lsst.multi INFO: cancelling 17194960 on provider astro 2024-03-18 05:19:26 proc_lsst.multi:146 [INFO] found job 17194961 in provider astro proc_lsst.multi INFO: found job 17194961 in provider astro 2024-03-18 05:19:26 proc_lsst.multi:201 [INFO] cancelling 17194961 on provider astro proc_lsst.multi INFO: cancelling 17194961 on provider astro 2024-03-18 05:19:26 proc_lsst.multi:146 [INFO] found job 17194962 in provider astro proc_lsst.multi INFO: found job 17194962 in provider astro 2024-03-18 05:19:26 proc_lsst.multi:201 [INFO] cancelling 17194962 on provider astro proc_lsst.multi INFO: cancelling 17194962 on provider astro 2024-03-18 05:19:27 proc_lsst.multi:146 [INFO] found job 17194963 in provider astro proc_lsst.multi INFO: found job 17194963 in provider astro 2024-03-18 05:19:27 proc_lsst.multi:201 [INFO] cancelling 17194963 on provider astro proc_lsst.multi INFO: cancelling 17194963 on provider astro 2024-03-18 05:19:27 proc_lsst.multi:146 [INFO] found job 17194964 in provider astro proc_lsst.multi INFO: found job 17194964 in provider astro 2024-03-18 05:19:27 proc_lsst.multi:201 [INFO] cancelling 17194964 on provider astro proc_lsst.multi INFO: cancelling 17194964 on provider astro 2024-03-18 05:19:27 proc_lsst.multi:146 [INFO] found job 17194965 in provider astro proc_lsst.multi INFO: found job 17194965 in provider astro 2024-03-18 05:19:27 proc_lsst.multi:201 [INFO] cancelling 17194965 on provider astro proc_lsst.multi INFO: cancelling 17194965 on provider astro 2024-03-18 05:19:27 proc_lsst.multi:146 [INFO] found job 17194966 in provider astro proc_lsst.multi INFO: found job 17194966 in provider astro 2024-03-18 05:19:27 proc_lsst.multi:201 [INFO] cancelling 17194966 on provider astro proc_lsst.multi INFO: cancelling 17194966 on provider astro 2024-03-18 05:19:27 proc_lsst.multi:146 [INFO] found job 17194967 in provider astro proc_lsst.multi INFO: found job 17194967 in provider astro 2024-03-18 05:19:27 proc_lsst.multi:201 [INFO] cancelling 17194967 on provider astro proc_lsst.multi INFO: cancelling 17194967 on provider astro 2024-03-18 05:19:27 proc_lsst.multi:146 [INFO] found job 17194968 in provider astro proc_lsst.multi INFO: found job 17194968 in provider astro 2024-03-18 05:19:27 proc_lsst.multi:201 [INFO] cancelling 17194968 on provider astro proc_lsst.multi INFO: cancelling 17194968 on provider astro 2024-03-18 05:19:27 proc_lsst.multi:146 [INFO] found job 17194969 in provider astro proc_lsst.multi INFO: found job 17194969 in provider astro 2024-03-18 05:19:27 proc_lsst.multi:201 [INFO] cancelling 17194969 on provider astro proc_lsst.multi INFO: cancelling 17194969 on provider astro 2024-03-18 05:19:27 proc_lsst.multi:146 [INFO] found job 17194970 in provider astro proc_lsst.multi INFO: found job 17194970 in provider astro 2024-03-18 05:19:27 proc_lsst.multi:201 [INFO] cancelling 17194970 on provider astro proc_lsst.multi INFO: cancelling 17194970 on provider astro 2024-03-18 05:19:27 proc_lsst.multi:146 [INFO] found job 17194971 in provider astro proc_lsst.multi INFO: found job 17194971 in provider astro 2024-03-18 05:19:27 proc_lsst.multi:201 [INFO] cancelling 17194971 on provider astro proc_lsst.multi INFO: cancelling 17194971 on provider astro 2024-03-18 05:19:27 proc_lsst.multi:146 [INFO] found job 17194973 in provider astro proc_lsst.multi INFO: found job 17194973 in provider astro 2024-03-18 05:19:27 proc_lsst.multi:201 [INFO] cancelling 17194973 on provider astro proc_lsst.multi INFO: cancelling 17194973 on provider astro 2024-03-18 05:19:28 proc_lsst.multi:146 [INFO] found job 17194974 in provider astro proc_lsst.multi INFO: found job 17194974 in provider astro 2024-03-18 05:19:28 proc_lsst.multi:201 [INFO] cancelling 17194974 on provider astro proc_lsst.multi INFO: cancelling 17194974 on provider astro 2024-03-18 05:19:28 proc_lsst.multi:146 [INFO] found job 17194975 in provider astro proc_lsst.multi INFO: found job 17194975 in provider astro 2024-03-18 05:19:28 proc_lsst.multi:201 [INFO] cancelling 17194975 on provider astro proc_lsst.multi INFO: cancelling 17194975 on provider astro 2024-03-18 05:19:28 proc_lsst.multi:146 [INFO] found job 17194976 in provider astro proc_lsst.multi INFO: found job 17194976 in provider astro 2024-03-18 05:19:28 proc_lsst.multi:201 [INFO] cancelling 17194976 on provider astro proc_lsst.multi INFO: cancelling 17194976 on provider astro 2024-03-18 05:19:28 proc_lsst.multi:146 [INFO] found job 17194977 in provider astro proc_lsst.multi INFO: found job 17194977 in provider astro 2024-03-18 05:19:28 proc_lsst.multi:201 [INFO] cancelling 17194977 on provider astro proc_lsst.multi INFO: cancelling 17194977 on provider astro 2024-03-18 05:19:28 proc_lsst.multi:146 [INFO] found job 17194978 in provider astro proc_lsst.multi INFO: found job 17194978 in provider astro 2024-03-18 05:19:28 proc_lsst.multi:201 [INFO] cancelling 17194978 on provider astro proc_lsst.multi INFO: cancelling 17194978 on provider astro 2024-03-18 05:19:28 proc_lsst.multi:146 [INFO] found job 17194979 in provider astro proc_lsst.multi INFO: found job 17194979 in provider astro 2024-03-18 05:19:28 proc_lsst.multi:201 [INFO] cancelling 17194979 on provider astro proc_lsst.multi INFO: cancelling 17194979 on provider astro 2024-03-18 05:19:28 proc_lsst.multi:146 [INFO] found job 17194980 in provider astro proc_lsst.multi INFO: found job 17194980 in provider astro 2024-03-18 05:19:28 proc_lsst.multi:201 [INFO] cancelling 17194980 on provider astro proc_lsst.multi INFO: cancelling 17194980 on provider astro 2024-03-18 05:19:28 proc_lsst.multi:146 [INFO] found job 17194981 in provider astro proc_lsst.multi INFO: found job 17194981 in provider astro 2024-03-18 05:19:28 proc_lsst.multi:201 [INFO] cancelling 17194981 on provider astro proc_lsst.multi INFO: cancelling 17194981 on provider astro 2024-03-18 05:19:28 proc_lsst.multi:146 [INFO] found job 17194982 in provider astro proc_lsst.multi INFO: found job 17194982 in provider astro 2024-03-18 05:19:28 proc_lsst.multi:201 [INFO] cancelling 17194982 on provider astro proc_lsst.multi INFO: cancelling 17194982 on provider astro 2024-03-18 05:19:29 proc_lsst.multi:146 [INFO] found job 17194983 in provider astro proc_lsst.multi INFO: found job 17194983 in provider astro 2024-03-18 05:19:29 proc_lsst.multi:201 [INFO] cancelling 17194983 on provider astro proc_lsst.multi INFO: cancelling 17194983 on provider astro 2024-03-18 05:19:29 proc_lsst.multi:146 [INFO] found job 17194989 in provider escience_a40 proc_lsst.multi INFO: found job 17194989 in provider escience_a40 2024-03-18 05:19:29 proc_lsst.multi:201 [INFO] cancelling 17194989 on provider escience_a40 proc_lsst.multi INFO: cancelling 17194989 on provider escience_a40 2024-03-18 05:19:29 proc_lsst.multi:146 [INFO] found job 17194990 in provider escience_a40 proc_lsst.multi INFO: found job 17194990 in provider escience_a40 2024-03-18 05:19:29 proc_lsst.multi:201 [INFO] cancelling 17194990 on provider escience_a40 proc_lsst.multi INFO: cancelling 17194990 on provider escience_a40 2024-03-18 05:19:29 proc_lsst.multi:146 [INFO] found job 17194991 in provider escience_a40 proc_lsst.multi INFO: found job 17194991 in provider escience_a40 2024-03-18 05:19:29 proc_lsst.multi:201 [INFO] cancelling 17194991 on provider escience_a40 proc_lsst.multi INFO: cancelling 17194991 on provider escience_a40 2024-03-18 05:19:29 proc_lsst.multi:146 [INFO] found job 17194992 in provider escience_a40 proc_lsst.multi INFO: found job 17194992 in provider escience_a40 2024-03-18 05:19:29 proc_lsst.multi:201 [INFO] cancelling 17194992 on provider escience_a40 proc_lsst.multi INFO: cancelling 17194992 on provider escience_a40 2024-03-18 05:19:29 proc_lsst.multi:146 [INFO] found job 17194993 in provider escience_a40 proc_lsst.multi INFO: found job 17194993 in provider escience_a40 2024-03-18 05:19:29 proc_lsst.multi:201 [INFO] cancelling 17194993 on provider escience_a40 proc_lsst.multi INFO: cancelling 17194993 on provider escience_a40 2024-03-18 05:19:29 proc_lsst.multi:146 [INFO] found job 17194994 in provider escience_a40 proc_lsst.multi INFO: found job 17194994 in provider escience_a40 2024-03-18 05:19:29 proc_lsst.multi:201 [INFO] cancelling 17194994 on provider escience_a40 proc_lsst.multi INFO: cancelling 17194994 on provider escience_a40 2024-03-18 05:19:29 proc_lsst.multi:146 [INFO] found job 17194995 in provider escience_a40 proc_lsst.multi INFO: found job 17194995 in provider escience_a40 2024-03-18 05:19:29 proc_lsst.multi:201 [INFO] cancelling 17194995 on provider escience_a40 proc_lsst.multi INFO: cancelling 17194995 on provider escience_a40 2024-03-18 05:19:29 proc_lsst.multi:146 [INFO] found job 17194996 in provider escience_a40 proc_lsst.multi INFO: found job 17194996 in provider escience_a40 2024-03-18 05:19:29 proc_lsst.multi:201 [INFO] cancelling 17194996 on provider escience_a40 proc_lsst.multi INFO: cancelling 17194996 on provider escience_a40 2024-03-18 05:19:30 proc_lsst.multi:146 [INFO] found job 17194997 in provider escience_a40 proc_lsst.multi INFO: found job 17194997 in provider escience_a40 2024-03-18 05:19:30 proc_lsst.multi:201 [INFO] cancelling 17194997 on provider escience_a40 proc_lsst.multi INFO: cancelling 17194997 on provider escience_a40 2024-03-18 05:19:30 proc_lsst.multi:146 [INFO] found job 17194998 in provider escience_a40 proc_lsst.multi INFO: found job 17194998 in provider escience_a40 2024-03-18 05:19:30 proc_lsst.multi:201 [INFO] cancelling 17194998 on provider escience_a40 proc_lsst.multi INFO: cancelling 17194998 on provider escience_a40 2024-03-18 05:19:30 proc_lsst.multi:146 [INFO] found job 17194999 in provider escience_a40 proc_lsst.multi INFO: found job 17194999 in provider escience_a40 2024-03-18 05:19:30 proc_lsst.multi:201 [INFO] cancelling 17194999 on provider escience_a40 proc_lsst.multi INFO: cancelling 17194999 on provider escience_a40 2024-03-18 05:19:30 proc_lsst.multi:146 [INFO] found job 17195000 in provider escience_a40 proc_lsst.multi INFO: found job 17195000 in provider escience_a40 2024-03-18 05:19:30 proc_lsst.multi:201 [INFO] cancelling 17195000 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195000 on provider escience_a40 2024-03-18 05:19:30 proc_lsst.multi:146 [INFO] found job 17195001 in provider escience_a40 proc_lsst.multi INFO: found job 17195001 in provider escience_a40 2024-03-18 05:19:30 proc_lsst.multi:201 [INFO] cancelling 17195001 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195001 on provider escience_a40 2024-03-18 05:19:30 proc_lsst.multi:146 [INFO] found job 17195002 in provider escience_a40 proc_lsst.multi INFO: found job 17195002 in provider escience_a40 2024-03-18 05:19:30 proc_lsst.multi:201 [INFO] cancelling 17195002 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195002 on provider escience_a40 2024-03-18 05:19:30 proc_lsst.multi:146 [INFO] found job 17195003 in provider escience_a40 proc_lsst.multi INFO: found job 17195003 in provider escience_a40 2024-03-18 05:19:30 proc_lsst.multi:201 [INFO] cancelling 17195003 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195003 on provider escience_a40 2024-03-18 05:19:30 proc_lsst.multi:146 [INFO] found job 17195004 in provider escience_a40 proc_lsst.multi INFO: found job 17195004 in provider escience_a40 2024-03-18 05:19:30 proc_lsst.multi:201 [INFO] cancelling 17195004 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195004 on provider escience_a40 2024-03-18 05:19:30 proc_lsst.multi:146 [INFO] found job 17195005 in provider escience_a40 proc_lsst.multi INFO: found job 17195005 in provider escience_a40 2024-03-18 05:19:30 proc_lsst.multi:201 [INFO] cancelling 17195005 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195005 on provider escience_a40 2024-03-18 05:19:30 proc_lsst.multi:146 [INFO] found job 17195006 in provider escience_a40 proc_lsst.multi INFO: found job 17195006 in provider escience_a40 2024-03-18 05:19:30 proc_lsst.multi:201 [INFO] cancelling 17195006 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195006 on provider escience_a40 2024-03-18 05:19:30 proc_lsst.multi:146 [INFO] found job 17195007 in provider escience_a40 proc_lsst.multi INFO: found job 17195007 in provider escience_a40 2024-03-18 05:19:30 proc_lsst.multi:201 [INFO] cancelling 17195007 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195007 on provider escience_a40 2024-03-18 05:19:30 proc_lsst.multi:146 [INFO] found job 17195008 in provider escience_a40 proc_lsst.multi INFO: found job 17195008 in provider escience_a40 2024-03-18 05:19:30 proc_lsst.multi:201 [INFO] cancelling 17195008 on provider escience_a40 proc_lsst.multi INFO: cancelling 17195008 on provider escience_a40 2024-03-18 05:19:31 proc_lsst.multi:146 [INFO] found job 17194984 in provider escience_rtx6k proc_lsst.multi INFO: found job 17194984 in provider escience_rtx6k 2024-03-18 05:19:31 proc_lsst.multi:201 [INFO] cancelling 17194984 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17194984 on provider escience_rtx6k 2024-03-18 05:19:31 proc_lsst.multi:146 [INFO] found job 17194985 in provider escience_rtx6k proc_lsst.multi INFO: found job 17194985 in provider escience_rtx6k 2024-03-18 05:19:31 proc_lsst.multi:201 [INFO] cancelling 17194985 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17194985 on provider escience_rtx6k 2024-03-18 05:19:31 proc_lsst.multi:146 [INFO] found job 17194986 in provider escience_rtx6k proc_lsst.multi INFO: found job 17194986 in provider escience_rtx6k 2024-03-18 05:19:31 proc_lsst.multi:201 [INFO] cancelling 17194986 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17194986 on provider escience_rtx6k 2024-03-18 05:19:31 proc_lsst.multi:146 [INFO] found job 17194987 in provider escience_rtx6k proc_lsst.multi INFO: found job 17194987 in provider escience_rtx6k 2024-03-18 05:19:31 proc_lsst.multi:201 [INFO] cancelling 17194987 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17194987 on provider escience_rtx6k 2024-03-18 05:19:31 proc_lsst.multi:146 [INFO] found job 17194988 in provider escience_rtx6k proc_lsst.multi INFO: found job 17194988 in provider escience_rtx6k 2024-03-18 05:19:31 proc_lsst.multi:201 [INFO] cancelling 17194988 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17194988 on provider escience_rtx6k 2024-03-18 05:19:31 proc_lsst.multi:146 [INFO] found job 53289 in provider local proc_lsst.multi INFO: found job 53289 in provider local 2024-03-18 05:19:31 proc_lsst.multi:201 [INFO] cancelling 53289 on provider local proc_lsst.multi INFO: cancelling 53289 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 53289 parsl.providers.local.local WARNING: Failed to kill PID: 53289 and child processes on local 2024-03-18 05:19:32 proc_lsst.multi:50 [INFO] no new jobs since last cancel, resuming executor shutdown proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt parsl.dataflow.dflow INFO: Shut down executor multi parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete parsl.executors.threads DEBUG: Done with executor shutdown parsl.dataflow.dflow INFO: Shut down executor _parsl_internal parsl.dataflow.dflow INFO: Terminated executors parsl.dataflow.dflow INFO: DFK cleanup complete parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 134.1270 seconds lsst.ctrl.bps.drivers INFO: Run 'DEEP_20211003_B1b_science#step0_20240318T051656Z' submitted for execution with id 'None' lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 134.1337 seconds; current memory usage: 0.277 Gibyte, delta: 0.003 Gibyte, peak delta: 0.005 Gibyte lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 147.4832 seconds; current memory usage: 0.277 Gibyte, delta: 0.138 Gibyte, peak delta: 0.140 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.279 Gibyte (main), 0.279 Gibyte (largest child process) Run Id: None Run Name: DEEP_20211003_B1b_science#step0_20240318T051656Z parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up real 2m29.344s user 0m38.623s sys 0m18.515s