+ PROC_LSST_SUBMIT_PATH=/mmfs1/home/stevengs/dirac/DEEP/submit + bps submit /gscratch/dirac/shared/opt/proc_lsst/pipelines/submit.yaml -b /mmfs1/home/stevengs/dirac/DEEP/repo -i DEEP/20190507/A1c --output-run DEEP/20190507/A1c/science#step0/20240309T204810Z --qgraph pipeline.qgraph lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process. lsst.ctrl.bps.drivers INFO: Starting submission process lsst.ctrl.bps.drivers INFO: Initializing execution environment lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 15.9096 seconds; current memory usage: 0.180 Gibyte, delta: 0.041 Gibyte, peak delta: 0.054 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.192 Gibyte (main), 0.000 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph) lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.3685 seconds lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.3599 seconds lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/pipeline_orig.qgraph DEEP/20190507/A1c/science#step0/20240309T204810Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/pipeline.qgraph lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 3.8179 seconds lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 11.4077 seconds; current memory usage: 0.274 Gibyte, delta: 0.094 Gibyte, peak delta: 0.082 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.274 Gibyte (main), 0.264 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs) lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0221 seconds; current memory usage: 0.274 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.274 Gibyte (main), 0.264 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 372 cluster(s) lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow) lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20190507_A1c_science#step0_20240309T204810Z' lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 1.1155 seconds; current memory usage: 0.274 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.274 Gibyte (main), 0.264 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 374 job(s) (including final) lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow) parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2 Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses s_addresses.add(address_by_interface(interface)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper retval = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface return socket.inet_ntoa(fcntl.ioctl( ^^^^^^^^^^^^ OSError: [Errno 99] Cannot assign requested address lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 16.2582 seconds; current memory usage: 0.276 Gibyte, delta: 0.001 Gibyte, peak delta: 0.001 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.276 Gibyte (main), 0.264 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting submit stage lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000 parsl.dataflow.dflow INFO: Starting DataFlowKernel with config Config( app_cache=True, checkpoint_files=None, checkpoint_mode='task_exit', checkpoint_period=None, executors=[MultiHighThroughputExecutor()], garbage_collect=True, initialize_logging=True, internal_tasks_max_threads=10, max_idletime=120.0, monitoring=None, retries=1, retry_handler=None, run_dir='runinfo', strategy='simple', usage_tracking=False ) parsl.dataflow.dflow INFO: Parsl version: 2023.06.12 parsl.usage_tracking.usage DEBUG: Tracking status: False parsl.dataflow.dflow INFO: Run id is: 26034a7f-325f-4787-840e-93b070288235 parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name parsl.dataflow.memoization INFO: App caching initialized parsl.dataflow.strategy DEBUG: Scaling strategy: simple parsl.executors.high_throughput.executor DEBUG: Starting queue management thread parsl.executors.high_throughput.executor DEBUG: queue management worker starting parsl.executors.high_throughput.executor DEBUG: Started queue management thread Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 0 executors parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22762785682704 parsl.executors.high_throughput.executor DEBUG: Created management thread: parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id={block_id} --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider: parsl.executors.status_handling INFO: Scaling out by 1 blocks parsl.executors.status_handling INFO: Allocated block ID 0 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0 2024-03-09 20:49:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 2024-03-09 20:49:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-09 20:49:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local parsl.providers.local.local DEBUG: Launching in remote mode 2024-03-09 20:49:33 proc_lsst.multi:170 [INFO] [multi] job_id 35332 proc_lsst.multi INFO: [multi] job_id 35332 2024-03-09 20:49:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:33 proc_lsst.multi:178 [INFO] [multi] provider local accepted submit and returned 35332 proc_lsst.multi INFO: [multi] provider local accepted submit and returned 35332 parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 35332 parsl.dataflow.job_status_poller DEBUG: Adding executor multi parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 0 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 0 has memoization hash 9730308ce7b3fd1d9b8a1a0d8c453774 parsl.dataflow.memoization INFO: Task 0 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de0053a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1 parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/bfc14203-d22c-46d9-bf29-c6a347b53121_isrForCrosstalkSources_855820_51.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/bfc14203-d22c-46d9-bf29-c6a347b53121_isrForCrosstalkSources_855820_51.stderr parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1 has memoization hash 23c1b35578eaab278162ebb1e53fde7d parsl.dataflow.memoization INFO: Task 1 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de0056c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 2 parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/ebe1783d-e57c-4518-aaa3-9a1b903e8f26_isrForCrosstalkSources_855909_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/ebe1783d-e57c-4518-aaa3-9a1b903e8f26_isrForCrosstalkSources_855909_25.stderr parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2 has memoization hash ca3347482424541120dec6dd8eeb2115 parsl.dataflow.memoization INFO: Task 2 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de0059e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 3 parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/c858dc63-c41e-4b23-acc7-af08f3000ede_isrForCrosstalkSources_855821_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/c858dc63-c41e-4b23-acc7-af08f3000ede_isrForCrosstalkSources_855821_49.stderr parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3 has memoization hash 6242cb7c13f59e88578c9f03737a0619 parsl.dataflow.memoization INFO: Task 3 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de0058a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 4 parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/7163b6fe-9d83-4149-a098-4330052fbdc6_isrForCrosstalkSources_855820_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/7163b6fe-9d83-4149-a098-4330052fbdc6_isrForCrosstalkSources_855820_53.stderr parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4 has memoization hash dfbc2c30ce8604a5515ae1a8f5753216 parsl.dataflow.memoization INFO: Task 4 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de005b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 5 parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/e47bdb68-8e57-40ed-9d4b-bd93f061788e_isrForCrosstalkSources_855911_14.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/e47bdb68-8e57-40ed-9d4b-bd93f061788e_isrForCrosstalkSources_855911_14.stderr parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5 has memoization hash bb58569d2ec10e135d7742fba5f2cd08 parsl.dataflow.memoization INFO: Task 5 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de005940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 6 parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/629bff43-6703-4a2c-bcb0-18e4dc9236b7_isrForCrosstalkSources_855911_52.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/629bff43-6703-4a2c-bcb0-18e4dc9236b7_isrForCrosstalkSources_855911_52.stderr parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 6 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 6 has memoization hash 7d86293cc63517cb6161795e2e7db631 parsl.dataflow.memoization INFO: Task 6 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de005bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 7 parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/52dc29e6-dbd9-426f-b32f-36a66300c4a4_isrForCrosstalkSources_855910_62.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/52dc29e6-dbd9-426f-b32f-36a66300c4a4_isrForCrosstalkSources_855910_62.stderr parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 7 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 7 has memoization hash 1d2cc8e21270e09daaeda34b74d92a80 parsl.dataflow.memoization INFO: Task 7 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de005260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 8 parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/4323d15f-d656-4997-a51f-1077a1434b6a_isrForCrosstalkSources_855909_16.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/4323d15f-d656-4997-a51f-1077a1434b6a_isrForCrosstalkSources_855909_16.stderr parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 8 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 8 has memoization hash 6bf303c81925fe2248459f0b57a4ea81 parsl.dataflow.memoization INFO: Task 8 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de005da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 9 parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/7b3be9a2-165e-4dce-91bb-f952df3371a5_isrForCrosstalkSources_855911_40.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/7b3be9a2-165e-4dce-91bb-f952df3371a5_isrForCrosstalkSources_855911_40.stderr parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 9 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 9 has memoization hash ac6f9f25b009df11f168f3e162b7eaca parsl.dataflow.memoization INFO: Task 9 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de005580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 10 parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/b3b73630-8bf7-4e1d-a834-6a3742312723_isrForCrosstalkSources_855821_1.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/b3b73630-8bf7-4e1d-a834-6a3742312723_isrForCrosstalkSources_855821_1.stderr parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 10 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 10 has memoization hash d89c5e967d77bc7b06e10bf5f89b34d1 parsl.dataflow.memoization INFO: Task 10 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de005a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 11 parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/75bb30b4-1d37-4269-8645-1e6d97cce4d7_isrForCrosstalkSources_855821_15.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/75bb30b4-1d37-4269-8645-1e6d97cce4d7_isrForCrosstalkSources_855821_15.stderr parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 11 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 11 has memoization hash 036cf9bd681733b8b49113807ba9bcd6 parsl.dataflow.memoization INFO: Task 11 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de005d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 12 parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/98ccd585-bded-47d6-8dee-4dc6c7095969_isrForCrosstalkSources_855822_61.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/98ccd585-bded-47d6-8dee-4dc6c7095969_isrForCrosstalkSources_855822_61.stderr parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 12 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 12 has memoization hash 84b03dedba2486c71bae636256bd15fa parsl.dataflow.memoization INFO: Task 12 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de006020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 13 parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/298a1c83-1958-433b-8a7b-bcf31af54664_isrForCrosstalkSources_855910_8.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/298a1c83-1958-433b-8a7b-bcf31af54664_isrForCrosstalkSources_855910_8.stderr parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 13 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 13 has memoization hash d5aac479bcb78ee241995d93af9a7e39 parsl.dataflow.memoization INFO: Task 13 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de006200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 14 parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/e8ef6668-3629-47b8-a379-d49c8a229a4d_isrForCrosstalkSources_855821_38.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/e8ef6668-3629-47b8-a379-d49c8a229a4d_isrForCrosstalkSources_855821_38.stderr parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 14 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 14 has memoization hash 8bc15986828408dcd816487ab51db7d8 parsl.dataflow.memoization INFO: Task 14 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de0063e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 15 parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/7f5c489b-702e-45be-9b11-83592dd1223a_isrForCrosstalkSources_855821_32.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/7f5c489b-702e-45be-9b11-83592dd1223a_isrForCrosstalkSources_855821_32.stderr parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 15 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 15 has memoization hash 959040d87b41e5c35a8047c9906a0be0 parsl.dataflow.memoization INFO: Task 15 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de005f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 16 parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/80c5f238-c4b4-4429-a37a-070721ae86e3_isrForCrosstalkSources_855909_32.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/80c5f238-c4b4-4429-a37a-070721ae86e3_isrForCrosstalkSources_855909_32.stderr parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 16 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 16 has memoization hash c309d6dc04bdaf8774df3254de1e5e20 parsl.dataflow.memoization INFO: Task 16 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de006480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 17 parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/11127bbd-bf4e-49ae-b993-3e06f77162d7_isrForCrosstalkSources_855909_7.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/11127bbd-bf4e-49ae-b993-3e06f77162d7_isrForCrosstalkSources_855909_7.stderr parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 17 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 17 has memoization hash c6da75f7f4513e224ede3ae9f84c3102 parsl.dataflow.memoization INFO: Task 17 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de006660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 18 parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/6f9b40db-1d33-4c2b-8bf9-c3e0670a16dd_isrForCrosstalkSources_855822_50.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/6f9b40db-1d33-4c2b-8bf9-c3e0670a16dd_isrForCrosstalkSources_855822_50.stderr parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 18 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 18 has memoization hash cc0c8e359f9ba53d71038675109e8255 parsl.dataflow.memoization INFO: Task 18 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de006520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 19 parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/9d5feb91-1328-4bd0-a1b5-5e068259b1bd_isrForCrosstalkSources_855910_9.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/9d5feb91-1328-4bd0-a1b5-5e068259b1bd_isrForCrosstalkSources_855910_9.stderr parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 19 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 19 has memoization hash 94849db646166a04a70626d85ebf7ac9 parsl.dataflow.memoization INFO: Task 19 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de0067a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 20 parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/7f3413a8-6f44-41df-97bb-f77f8d03d958_isrForCrosstalkSources_855821_33.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/7f3413a8-6f44-41df-97bb-f77f8d03d958_isrForCrosstalkSources_855821_33.stderr parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 20 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 20 has memoization hash 99f8fb6a49f033ad134ee81d357de2fc parsl.dataflow.memoization INFO: Task 20 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de006340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 21 parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/f46496b6-2ebe-438d-8ea7-4eecdedec3cd_isrForCrosstalkSources_855910_45.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/f46496b6-2ebe-438d-8ea7-4eecdedec3cd_isrForCrosstalkSources_855910_45.stderr parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 21 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 21 has memoization hash e0e283f00f22fb203b6ebf316a51554f parsl.dataflow.memoization INFO: Task 21 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de006700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 22 parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/410cbe8d-858c-4405-8128-d33cec152849_isrForCrosstalkSources_855910_16.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/410cbe8d-858c-4405-8128-d33cec152849_isrForCrosstalkSources_855910_16.stderr parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 22 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash ecc38ec7eb2da95c4e07faa561e3e944 parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de0068e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 23 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/3f0fc7fd-e47c-4837-86a2-68a4058c16bd_isrForCrosstalkSources_855822_54.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/3f0fc7fd-e47c-4837-86a2-68a4058c16bd_isrForCrosstalkSources_855822_54.stderr parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 23 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 23 has memoization hash 912c09d72f49ca21451454aa9adef074 parsl.dataflow.memoization INFO: Task 23 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de006840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 23 try 0 launched on executor multi with executor id 24 parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/c4ce1a06-2b31-4a8c-8339-d5c6114593fc_isrForCrosstalkSources_855821_35.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/c4ce1a06-2b31-4a8c-8339-d5c6114593fc_isrForCrosstalkSources_855821_35.stderr parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 24 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 24 has memoization hash b8264e3d221f66310ad6b11250f5d1b6 parsl.dataflow.memoization INFO: Task 24 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de006ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 25 parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/639ad00f-743d-4cc6-a43d-5b8e5c267243_isrForCrosstalkSources_855910_5.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/639ad00f-743d-4cc6-a43d-5b8e5c267243_isrForCrosstalkSources_855910_5.stderr parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 25 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 25 has memoization hash a2ef8b79c8ea3648fd020b4d93f38b2e parsl.dataflow.memoization INFO: Task 25 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de006a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 26 parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/9924596f-d526-4763-ac2d-b385a6d93d59_isrForCrosstalkSources_855820_21.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/9924596f-d526-4763-ac2d-b385a6d93d59_isrForCrosstalkSources_855820_21.stderr parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 26 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 26 has memoization hash df82657af979c3406712f920a7746eea parsl.dataflow.memoization INFO: Task 26 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de006980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 27 parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/7d181afa-c2fd-458c-af9b-4a318478141a_isrForCrosstalkSources_855820_34.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/7d181afa-c2fd-458c-af9b-4a318478141a_isrForCrosstalkSources_855820_34.stderr parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 27 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 27 has memoization hash 5c4ed15b3d5ad4a562647043d43932f3 parsl.dataflow.memoization INFO: Task 27 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de006c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 28 parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/a5753eda-1820-47e4-9316-66eedbb7b152_isrForCrosstalkSources_855910_2.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/a5753eda-1820-47e4-9316-66eedbb7b152_isrForCrosstalkSources_855910_2.stderr parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 28 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash 51f8abf8b955e7f3338c8f92b5f06d4a parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de006ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 29 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/d152e696-0526-469b-bc08-9367bc01738e_isrForCrosstalkSources_855909_59.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/d152e696-0526-469b-bc08-9367bc01738e_isrForCrosstalkSources_855909_59.stderr parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 29 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 29 has memoization hash 4614c01a213acdfa0ebd1b11a94e9640 parsl.dataflow.memoization INFO: Task 29 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de006b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 29 try 0 launched on executor multi with executor id 30 parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/4eca3972-92d8-47bd-bc84-7a4368944aa0_isrForCrosstalkSources_855910_23.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/4eca3972-92d8-47bd-bc84-7a4368944aa0_isrForCrosstalkSources_855910_23.stderr parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 30 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 30 has memoization hash 5680d40fea48b98731d3445801ffdbc3 parsl.dataflow.memoization INFO: Task 30 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de006d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 31 parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/44c2a1b8-3d1a-4acd-a3c2-006c83c82a25_isrForCrosstalkSources_855911_10.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/44c2a1b8-3d1a-4acd-a3c2-006c83c82a25_isrForCrosstalkSources_855911_10.stderr parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 31 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 31 has memoization hash d81cea9eda84a0ba83ac0db5bc746b37 parsl.dataflow.memoization INFO: Task 31 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de006e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 32 parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/edde4809-693c-4bbc-8485-2b898f9b828c_isrForCrosstalkSources_855820_6.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/edde4809-693c-4bbc-8485-2b898f9b828c_isrForCrosstalkSources_855820_6.stderr parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 32 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 32 has memoization hash 91130eb60f71de87c949f7e2a600293c parsl.dataflow.memoization INFO: Task 32 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de006f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 33 parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/6462b175-d74b-47a7-a294-a00ef52d2355_isrForCrosstalkSources_855820_25.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/6462b175-d74b-47a7-a294-a00ef52d2355_isrForCrosstalkSources_855820_25.stderr parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 33 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 33 has memoization hash 5f639c0f7268e9728f72df3099962ead parsl.dataflow.memoization INFO: Task 33 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de006fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 34 parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/47625e6d-01c1-41c9-986d-324e9dac5252_isrForCrosstalkSources_855910_54.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/47625e6d-01c1-41c9-986d-324e9dac5252_isrForCrosstalkSources_855910_54.stderr parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 34 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash b3466ec710de251ef5645b608c8548b2 parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de007060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 35 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/529dfbe1-963d-44c0-b886-7f39d26d5a9a_isrForCrosstalkSources_855909_2.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/529dfbe1-963d-44c0-b886-7f39d26d5a9a_isrForCrosstalkSources_855909_2.stderr parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 35 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 35 has memoization hash 28646f34abf5723fc45f23d26ca46815 parsl.dataflow.memoization INFO: Task 35 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de006de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 35 try 0 launched on executor multi with executor id 36 parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/833a5e7d-bb7f-4654-b3f2-534e2cb0f5f2_isrForCrosstalkSources_855821_13.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/833a5e7d-bb7f-4654-b3f2-534e2cb0f5f2_isrForCrosstalkSources_855821_13.stderr parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 36 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 36 has memoization hash 385f7f558af3f8269f48516396ec317c parsl.dataflow.memoization INFO: Task 36 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de007100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 37 parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/d5aa8fd0-79f8-49b1-9da9-e1f8120d32f6_isrForCrosstalkSources_855910_59.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/d5aa8fd0-79f8-49b1-9da9-e1f8120d32f6_isrForCrosstalkSources_855910_59.stderr parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 37 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 37 has memoization hash 5d9f6f99b8108271c22fd8f7bebece5c parsl.dataflow.memoization INFO: Task 37 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de007240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 38 parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/9eba03a8-b3fb-40fe-8d94-d29bbf9cb8a6_isrForCrosstalkSources_855910_60.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/9eba03a8-b3fb-40fe-8d94-d29bbf9cb8a6_isrForCrosstalkSources_855910_60.stderr parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 38 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 38 has memoization hash af27a55e03fc324ce04031ae5e05d726 parsl.dataflow.memoization INFO: Task 38 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de007f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 39 parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/1094d02b-afab-41cc-a7cb-29248d53425a_isrForCrosstalkSources_855822_59.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/1094d02b-afab-41cc-a7cb-29248d53425a_isrForCrosstalkSources_855822_59.stderr parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 39 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 39 has memoization hash 13a993332896179fd63f60d8833fdba2 parsl.dataflow.memoization INFO: Task 39 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de007ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 40 parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/02abf915-1523-4e31-b275-b6af998a9f36_isrForCrosstalkSources_855911_26.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/02abf915-1523-4e31-b275-b6af998a9f36_isrForCrosstalkSources_855911_26.stderr parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 40 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 40 has memoization hash f84a128407667e6d7b447511420d0507 parsl.dataflow.memoization INFO: Task 40 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de007e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 41 parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/70b25e75-a62b-482d-aa80-7e4f80e6d73c_isrForCrosstalkSources_855821_11.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/70b25e75-a62b-482d-aa80-7e4f80e6d73c_isrForCrosstalkSources_855821_11.stderr parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 41 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 41 has memoization hash 1505663db626bc79340ba206e4c23b9d parsl.dataflow.memoization INFO: Task 41 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de007d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 42 parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/9acadd6d-dfe8-4cb4-9735-41ae9a0612fc_isrForCrosstalkSources_855820_23.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/9acadd6d-dfe8-4cb4-9735-41ae9a0612fc_isrForCrosstalkSources_855820_23.stderr parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 42 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 42 has memoization hash 2e707037e3b43820aea58e5afa1c387d parsl.dataflow.memoization INFO: Task 42 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de007ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 43 parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/5560782f-47d9-4559-80d8-5c9c55dbbdef_isrForCrosstalkSources_855909_40.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/5560782f-47d9-4559-80d8-5c9c55dbbdef_isrForCrosstalkSources_855909_40.stderr parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 43 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 43 has memoization hash dbc6c36c106f7de821fe7f3a11d91084 parsl.dataflow.memoization INFO: Task 43 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de007c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 44 parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/1bd881f6-c1fd-49fb-b6fb-8c238b0ee0ee_isrForCrosstalkSources_855911_15.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/1bd881f6-c1fd-49fb-b6fb-8c238b0ee0ee_isrForCrosstalkSources_855911_15.stderr parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 44 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 44 has memoization hash 40fbcc6b5b421f240b52913f18a84fb5 parsl.dataflow.memoization INFO: Task 44 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de007b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 45 parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/0e70ddd2-f6f0-4b3c-a962-0201690d3ef0_isrForCrosstalkSources_855821_26.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/0e70ddd2-f6f0-4b3c-a962-0201690d3ef0_isrForCrosstalkSources_855821_26.stderr parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 45 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 45 has memoization hash e15c09b6ca71f06070982502641ba778 parsl.dataflow.memoization INFO: Task 45 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de007ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 46 parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/1fd3dcf8-34a9-4753-a5fb-ae5ab6d8b634_isrForCrosstalkSources_855822_12.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/1fd3dcf8-34a9-4753-a5fb-ae5ab6d8b634_isrForCrosstalkSources_855822_12.stderr parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 46 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 46 has memoization hash 24a453709e9bdd3abfe45ea541f531ec parsl.dataflow.memoization INFO: Task 46 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de007a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 47 parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/9008735a-ebdb-48da-9ade-c983df0319e1_isrForCrosstalkSources_855820_41.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/9008735a-ebdb-48da-9ade-c983df0319e1_isrForCrosstalkSources_855820_41.stderr parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 47 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 47 has memoization hash b42b7ecb7227c9aac74d5dddaffa72c5 parsl.dataflow.memoization INFO: Task 47 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de0079c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 48 parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/2419087c-1e24-4307-9127-639c5e0c2895_isrForCrosstalkSources_855821_54.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/2419087c-1e24-4307-9127-639c5e0c2895_isrForCrosstalkSources_855821_54.stderr parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 48 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 48 has memoization hash 5b7693a2bc1ffdffe251decd97188ebf parsl.dataflow.memoization INFO: Task 48 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de007920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 49 parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/464fdfa7-9995-493e-a108-5b0561680ed9_isrForCrosstalkSources_855909_53.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/464fdfa7-9995-493e-a108-5b0561680ed9_isrForCrosstalkSources_855909_53.stderr parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 49 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 49 has memoization hash 8056ea5faf2c9a1728157671fbcaa6f5 parsl.dataflow.memoization INFO: Task 49 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de0077e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 50 parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/ff960574-fef0-42a0-9498-fdb371d2639e_isrForCrosstalkSources_855911_4.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/ff960574-fef0-42a0-9498-fdb371d2639e_isrForCrosstalkSources_855911_4.stderr parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 50 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 50 has memoization hash 0beb5af27886ff936b4be33ca17a28c6 parsl.dataflow.memoization INFO: Task 50 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de007880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 51 parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/643159a2-4de9-46ec-b38c-dc8cd97ed166_isrForCrosstalkSources_855911_35.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/643159a2-4de9-46ec-b38c-dc8cd97ed166_isrForCrosstalkSources_855911_35.stderr parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 51 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 51 has memoization hash 69b8dd7af1af9467938ef540e23b9352 parsl.dataflow.memoization INFO: Task 51 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de007740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 52 parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/2ba47074-caff-40e8-8997-4cb02487999f_isrForCrosstalkSources_855820_57.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/2ba47074-caff-40e8-8997-4cb02487999f_isrForCrosstalkSources_855820_57.stderr parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 52 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash 4653d266ce2589cb29a454a11de0837e parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de0076a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 53 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/642b7cf9-266c-4336-8de4-3a63dd814f26_isrForCrosstalkSources_855911_31.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/642b7cf9-266c-4336-8de4-3a63dd814f26_isrForCrosstalkSources_855911_31.stderr parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 53 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 53 has memoization hash ec4e3f117df1a44280f910a2706c153d parsl.dataflow.memoization INFO: Task 53 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de007600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 53 try 0 launched on executor multi with executor id 54 parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/7e8d853c-b34c-47fb-9ef5-cb981805b8fc_isrForCrosstalkSources_855822_42.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/7e8d853c-b34c-47fb-9ef5-cb981805b8fc_isrForCrosstalkSources_855822_42.stderr parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 54 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 54 has memoization hash 84fec48006d241a2e940ed4d8e96bf46 parsl.dataflow.memoization INFO: Task 54 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de0074c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 55 parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/e8b6ca86-849a-4c05-86bd-9e21963a50a5_isrForCrosstalkSources_855911_60.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/e8b6ca86-849a-4c05-86bd-9e21963a50a5_isrForCrosstalkSources_855911_60.stderr parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 55 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 55 has memoization hash 202e76aca9b27d00bf4c350413cbe509 parsl.dataflow.memoization INFO: Task 55 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de007560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 56 parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/a996961c-3b7c-450c-aefd-a2e9694bc088_isrForCrosstalkSources_855822_21.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/a996961c-3b7c-450c-aefd-a2e9694bc088_isrForCrosstalkSources_855822_21.stderr parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 56 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 56 has memoization hash 4431a88b1f340d8c4dd0e6dbd1e80cab parsl.dataflow.memoization INFO: Task 56 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de007420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 57 parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/3ae32828-cdee-4d68-b1b0-c4add05e55c8_isrForCrosstalkSources_855820_49.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/3ae32828-cdee-4d68-b1b0-c4add05e55c8_isrForCrosstalkSources_855820_49.stderr parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 57 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 57 has memoization hash 35608d00d936501bd90932f76ff7f69b parsl.dataflow.memoization INFO: Task 57 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de007380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 58 parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/1148a7d7-e1d9-4b3f-8f51-290ade855724_isrForCrosstalkSources_855822_39.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/1148a7d7-e1d9-4b3f-8f51-290ade855724_isrForCrosstalkSources_855822_39.stderr parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 58 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 58 has memoization hash 7d67780a64de113b54a34821b018586a parsl.dataflow.memoization INFO: Task 58 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de0065c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 59 parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/4224aa25-7978-4647-955b-dd07f371b6e1_isrForCrosstalkSources_855820_7.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/4224aa25-7978-4647-955b-dd07f371b6e1_isrForCrosstalkSources_855820_7.stderr parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 59 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 59 has memoization hash caaeab4ee9da5ce348687e143f0bfdda parsl.dataflow.memoization INFO: Task 59 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de0071a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 60 parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/92b66c65-3c16-423b-b29d-e40ca2a43f35_isrForCrosstalkSources_855821_51.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/92b66c65-3c16-423b-b29d-e40ca2a43f35_isrForCrosstalkSources_855821_51.stderr parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 60 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 60 has memoization hash a4daf2ce3b468d956243e38cf4c97560 parsl.dataflow.memoization INFO: Task 60 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de0400e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 61 parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/ed68e1a7-2c2c-4c27-98ad-49da54e517c1_isrForCrosstalkSources_855821_25.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/ed68e1a7-2c2c-4c27-98ad-49da54e517c1_isrForCrosstalkSources_855821_25.stderr parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 61 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 61 has memoization hash b978f558c971748ff6121f0c030ef347 parsl.dataflow.memoization INFO: Task 61 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de040180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 62 parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/f59fc9f3-c272-4eb1-af0e-05f629fa1b76_isrForCrosstalkSources_855911_46.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/f59fc9f3-c272-4eb1-af0e-05f629fa1b76_isrForCrosstalkSources_855911_46.stderr parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 62 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 62 has memoization hash fa6234532135f4d74db6729771036892 parsl.dataflow.memoization INFO: Task 62 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de040220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 63 parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/b2edaffd-33ea-4401-ba65-20c9e33d7113_isrForCrosstalkSources_855911_61.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/b2edaffd-33ea-4401-ba65-20c9e33d7113_isrForCrosstalkSources_855911_61.stderr parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 63 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 63 has memoization hash 768283743806c2612ced52142bb4f83b parsl.dataflow.memoization INFO: Task 63 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de040360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 64 parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/b642830d-983c-4910-a214-ca1aa987509f_isrForCrosstalkSources_855820_62.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/b642830d-983c-4910-a214-ca1aa987509f_isrForCrosstalkSources_855820_62.stderr parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 64 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 64 has memoization hash 376b5bdb6545df8444a3f0542837f9bd parsl.dataflow.memoization INFO: Task 64 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de0402c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 65 parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/8920c2bc-e237-45b2-a575-ce1b1060cb9d_isrForCrosstalkSources_855911_45.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/8920c2bc-e237-45b2-a575-ce1b1060cb9d_isrForCrosstalkSources_855911_45.stderr parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 65 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 65 has memoization hash d60b087b5b5f3d953eba9de87a195711 parsl.dataflow.memoization INFO: Task 65 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de040400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 65 try 0 launched on executor multi with executor id 66 parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/131f4075-4513-4fe6-af03-0a542bed621e_isrForCrosstalkSources_855911_33.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/131f4075-4513-4fe6-af03-0a542bed621e_isrForCrosstalkSources_855911_33.stderr parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 66 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 66 has memoization hash e80ffd0e10754062bb8631cc7b75e5c1 parsl.dataflow.memoization INFO: Task 66 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de0404a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 67 parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/7e1085fa-a473-4cdc-897c-22c6d723aa59_isrForCrosstalkSources_855911_11.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/7e1085fa-a473-4cdc-897c-22c6d723aa59_isrForCrosstalkSources_855911_11.stderr parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 67 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 67 has memoization hash 4bbe324d76c278d004364795b83cc3ca parsl.dataflow.memoization INFO: Task 67 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de040540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 68 parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/b9e67966-1e72-42d1-97f8-6ffc03b8e2a4_isrForCrosstalkSources_855911_42.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/b9e67966-1e72-42d1-97f8-6ffc03b8e2a4_isrForCrosstalkSources_855911_42.stderr parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 68 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 68 has memoization hash d08f2c354c7bbc1f726617e5f7124de1 parsl.dataflow.memoization INFO: Task 68 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de040680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 69 parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/78895cae-6bff-426a-8e83-5a4dfabfa5f2_isrForCrosstalkSources_855911_16.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/78895cae-6bff-426a-8e83-5a4dfabfa5f2_isrForCrosstalkSources_855911_16.stderr parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 69 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 69 has memoization hash f19a346fe9dd6dbd859c6ca65f0cc2d7 parsl.dataflow.memoization INFO: Task 69 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de0405e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 70 parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/82e36f6e-b7cc-4ae3-b67b-15d6ebf09227_isrForCrosstalkSources_855909_54.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/82e36f6e-b7cc-4ae3-b67b-15d6ebf09227_isrForCrosstalkSources_855909_54.stderr parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 70 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 70 has memoization hash 415390b2b8fee6d4310a0982af1d8444 parsl.dataflow.memoization INFO: Task 70 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de040720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 71 parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/597e24b3-d429-4bcb-9659-eb7f9c7ba2f5_isrForCrosstalkSources_855820_56.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/597e24b3-d429-4bcb-9659-eb7f9c7ba2f5_isrForCrosstalkSources_855820_56.stderr parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 71 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 71 has memoization hash 554ecda2dd2f8b75aaee69be124eba97 parsl.dataflow.memoization INFO: Task 71 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de0407c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 71 try 0 launched on executor multi with executor id 72 parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/f934bc3d-0393-4300-af59-1f8428e7412f_isrForCrosstalkSources_855909_36.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/f934bc3d-0393-4300-af59-1f8428e7412f_isrForCrosstalkSources_855909_36.stderr parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 72 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 72 has memoization hash a850075b03dac8cbe0559b7285adca17 parsl.dataflow.memoization INFO: Task 72 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de040860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 73 parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/be0cfc9d-e3ff-4a47-af7d-ed26fe20df0b_isrForCrosstalkSources_855910_42.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/be0cfc9d-e3ff-4a47-af7d-ed26fe20df0b_isrForCrosstalkSources_855910_42.stderr parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 73 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 73 has memoization hash c62cc86e46d4efc7e8810f526925eda3 parsl.dataflow.memoization INFO: Task 73 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de0409a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 74 parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/46d28b07-69fd-4e9a-aac8-ee83845f8493_isrForCrosstalkSources_855822_30.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/46d28b07-69fd-4e9a-aac8-ee83845f8493_isrForCrosstalkSources_855822_30.stderr parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 74 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 74 has memoization hash 5aa7884cc0bd7a7a636b41789810f1b3 parsl.dataflow.memoization INFO: Task 74 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de040900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 75 parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/8b9850eb-befa-45b0-9184-b985ac830bd4_isrForCrosstalkSources_855820_38.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/8b9850eb-befa-45b0-9184-b985ac830bd4_isrForCrosstalkSources_855820_38.stderr parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 75 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 75 has memoization hash f99d286b182fa4b033f40b25b73385ca parsl.dataflow.memoization INFO: Task 75 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de040a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 76 parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/18c5d542-aed9-4caa-8381-f31e3d9b6c27_isrForCrosstalkSources_855820_12.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/18c5d542-aed9-4caa-8381-f31e3d9b6c27_isrForCrosstalkSources_855820_12.stderr parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 76 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash 7769508b06b563f13032287adab4c110 parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de040ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 77 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/4c3bd2cd-9569-4805-a375-b70d04370148_isrForCrosstalkSources_855909_30.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/4c3bd2cd-9569-4805-a375-b70d04370148_isrForCrosstalkSources_855909_30.stderr parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 77 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 77 has memoization hash de10a393bd01fc73c454b4090287caab parsl.dataflow.memoization INFO: Task 77 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de040b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 77 try 0 launched on executor multi with executor id 78 parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/e7c44aa8-34c2-4441-b956-1659a280d74f_isrForCrosstalkSources_855820_20.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/e7c44aa8-34c2-4441-b956-1659a280d74f_isrForCrosstalkSources_855820_20.stderr parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 78 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 78 has memoization hash 44d2b33123904841f7db9b2f5132aafc parsl.dataflow.memoization INFO: Task 78 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de040cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 79 parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/94d982f3-bf5c-4d8d-836a-2224bb0ccd0a_isrForCrosstalkSources_855911_27.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/94d982f3-bf5c-4d8d-836a-2224bb0ccd0a_isrForCrosstalkSources_855911_27.stderr parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 79 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 79 has memoization hash c6309f15117fe9cdd3e18de2a8fc17e4 parsl.dataflow.memoization INFO: Task 79 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de040c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 80 parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/4469e5fa-e768-4c49-ab9c-a305448d873d_isrForCrosstalkSources_855910_17.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/4469e5fa-e768-4c49-ab9c-a305448d873d_isrForCrosstalkSources_855910_17.stderr parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 80 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 80 has memoization hash 1b5966306eeac768a816ad14ab6bf8de parsl.dataflow.memoization INFO: Task 80 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de040d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 81 parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/bcb4dda4-98d2-44a5-a91e-ce820b3ae440_isrForCrosstalkSources_855910_44.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/bcb4dda4-98d2-44a5-a91e-ce820b3ae440_isrForCrosstalkSources_855910_44.stderr parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 81 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 81 has memoization hash 6b72c01347d9d580f4f17cedc487c7e5 parsl.dataflow.memoization INFO: Task 81 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de040e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 82 parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/4d4340d5-b095-4dd9-8a7c-fef548a7fac9_isrForCrosstalkSources_855911_59.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/4d4340d5-b095-4dd9-8a7c-fef548a7fac9_isrForCrosstalkSources_855911_59.stderr parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 82 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 82 has memoization hash 01aa5ca2157dfa8f2a4e6d5a07fac99f parsl.dataflow.memoization INFO: Task 82 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de040ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 83 parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/f1d9e07c-efbc-4dc6-aede-8300449ee33a_isrForCrosstalkSources_855820_5.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/f1d9e07c-efbc-4dc6-aede-8300449ee33a_isrForCrosstalkSources_855820_5.stderr parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 83 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 83 has memoization hash aaeaafa53e9b80d1dcddfe7864cb74a0 parsl.dataflow.memoization INFO: Task 83 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de040fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 83 try 0 launched on executor multi with executor id 84 parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/e94695f0-0e84-4fed-82d0-ab7d9ea10207_isrForCrosstalkSources_855822_18.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/e94695f0-0e84-4fed-82d0-ab7d9ea10207_isrForCrosstalkSources_855822_18.stderr parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 84 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 84 has memoization hash c565e8428dd0ca19a7c6d0c48c9391e7 parsl.dataflow.memoization INFO: Task 84 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de040f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 85 parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/160ba715-7ba2-40e0-acdd-35c81c3b5aaa_isrForCrosstalkSources_855910_22.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/160ba715-7ba2-40e0-acdd-35c81c3b5aaa_isrForCrosstalkSources_855910_22.stderr parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 85 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 85 has memoization hash bc439cf4fe347d9c51f6b302d1c45516 parsl.dataflow.memoization INFO: Task 85 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de041080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 86 parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/ca81d67f-d5a1-4267-9c29-358164dcbe46_isrForCrosstalkSources_855820_39.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/ca81d67f-d5a1-4267-9c29-358164dcbe46_isrForCrosstalkSources_855820_39.stderr parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 86 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 86 has memoization hash 0a47780112dd9fab791edaf37b3bbc08 parsl.dataflow.memoization INFO: Task 86 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de041120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 87 parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/d3ad82ff-4bd6-43fa-9f6c-af4a8c9681b4_isrForCrosstalkSources_855909_60.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/d3ad82ff-4bd6-43fa-9f6c-af4a8c9681b4_isrForCrosstalkSources_855909_60.stderr parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 87 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 87 has memoization hash 62f16fb7e20934a1dc2abdda6d01c5bf parsl.dataflow.memoization INFO: Task 87 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de0411c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 88 parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/b1105cc8-1b72-41ea-8fe2-09c6b3a0f242_isrForCrosstalkSources_855822_5.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/b1105cc8-1b72-41ea-8fe2-09c6b3a0f242_isrForCrosstalkSources_855822_5.stderr parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 88 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 88 has memoization hash 7e7af0eb634cd1c919cb61d3d00f7410 parsl.dataflow.memoization INFO: Task 88 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de041300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 89 parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/fe488e0c-208f-425b-9f7e-6f89b078cea6_isrForCrosstalkSources_855822_26.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/fe488e0c-208f-425b-9f7e-6f89b078cea6_isrForCrosstalkSources_855822_26.stderr parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 89 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 89 has memoization hash 33011f3b432635dca6450d43112961c5 parsl.dataflow.memoization INFO: Task 89 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de041260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 90 parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/a5180d8f-c3f0-4b68-828b-4c6214ff9657_isrForCrosstalkSources_855911_6.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/a5180d8f-c3f0-4b68-828b-4c6214ff9657_isrForCrosstalkSources_855911_6.stderr parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 90 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 90 has memoization hash ade40d9875fe99a883fd0832e624150f parsl.dataflow.memoization INFO: Task 90 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de0413a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 91 parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/5708f642-fbdb-4f57-9dd3-62a76bf926f7_isrForCrosstalkSources_855910_43.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/5708f642-fbdb-4f57-9dd3-62a76bf926f7_isrForCrosstalkSources_855910_43.stderr parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 91 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 91 has memoization hash 7bd7873d6c38c31e85428772afd5400b parsl.dataflow.memoization INFO: Task 91 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de041440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 91 try 0 launched on executor multi with executor id 92 parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/6c6e9c99-b821-43c5-9db7-b70e7cf8db82_isrForCrosstalkSources_855822_14.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/6c6e9c99-b821-43c5-9db7-b70e7cf8db82_isrForCrosstalkSources_855822_14.stderr parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 92 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 92 has memoization hash 9556a094525006bd09a58bd44e6c1b55 parsl.dataflow.memoization INFO: Task 92 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de0414e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 93 parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/1edddfda-8049-4327-992d-7e5543e27889_isrForCrosstalkSources_855910_10.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/1edddfda-8049-4327-992d-7e5543e27889_isrForCrosstalkSources_855910_10.stderr parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 93 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 93 has memoization hash 94fb5bfe4034cc96acbe1b82c5bfa31c parsl.dataflow.memoization INFO: Task 93 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de041620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 94 parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/e5b231ed-9851-41a7-8c3f-0a57f24d1c5a_isrForCrosstalkSources_855822_6.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/e5b231ed-9851-41a7-8c3f-0a57f24d1c5a_isrForCrosstalkSources_855822_6.stderr parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 94 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 94 has memoization hash 9c0b213cbbeb73deb9528374f1b038dd parsl.dataflow.memoization INFO: Task 94 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de041580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 95 parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/bd8cc624-8612-4c13-b0e3-706c88b6310b_isrForCrosstalkSources_855822_38.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/bd8cc624-8612-4c13-b0e3-706c88b6310b_isrForCrosstalkSources_855822_38.stderr parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 95 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 95 has memoization hash 5e9d773488bdbb080b8df98123fab84a parsl.dataflow.memoization INFO: Task 95 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de0416c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 96 parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/d740cb42-bf17-4d95-8836-c147ab3ac4f7_isrForCrosstalkSources_855911_2.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/d740cb42-bf17-4d95-8836-c147ab3ac4f7_isrForCrosstalkSources_855911_2.stderr parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 96 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 96 has memoization hash c017cb5e7db1781fed7f6eac2ea2a909 parsl.dataflow.memoization INFO: Task 96 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de041760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 97 parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/0af4fd2f-6336-421c-8660-ad747b62b8f6_isrForCrosstalkSources_855822_56.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/0af4fd2f-6336-421c-8660-ad747b62b8f6_isrForCrosstalkSources_855822_56.stderr parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 97 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 97 has memoization hash 1ea5342be757156fb198f513e61f131e parsl.dataflow.memoization INFO: Task 97 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de041800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 98 parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/caa0eb0a-0c43-43b7-8c3d-302f9cc823f2_isrForCrosstalkSources_855910_32.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/caa0eb0a-0c43-43b7-8c3d-302f9cc823f2_isrForCrosstalkSources_855910_32.stderr parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 98 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 98 has memoization hash b7f3df18c4fb4dd3c4437fe672bcaab7 parsl.dataflow.memoization INFO: Task 98 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de041940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 99 parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/e69b0fde-51a2-4394-bd40-c6eb2747fd0e_isrForCrosstalkSources_855820_17.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/e69b0fde-51a2-4394-bd40-c6eb2747fd0e_isrForCrosstalkSources_855820_17.stderr parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 99 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 99 has memoization hash 526259724f182762f55664c898432039 parsl.dataflow.memoization INFO: Task 99 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de0418a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 100 parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/7b8e8dad-27f4-4a0a-98ab-e6f18f09a98f_isrForCrosstalkSources_855909_33.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/7b8e8dad-27f4-4a0a-98ab-e6f18f09a98f_isrForCrosstalkSources_855909_33.stderr parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 100 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash 0a981057fe53e9a0982c6d00ea9ed4b0 parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de0419e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 101 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/194af422-dd03-447d-9358-072bb1685e12_isrForCrosstalkSources_855910_24.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/194af422-dd03-447d-9358-072bb1685e12_isrForCrosstalkSources_855910_24.stderr parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 101 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 101 has memoization hash 4b1a3e85b4071666aaac42cc10b41c39 parsl.dataflow.memoization INFO: Task 101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de041a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 101 try 0 launched on executor multi with executor id 102 parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/0f1443a5-950b-412e-b7ad-aa490d0a0898_isrForCrosstalkSources_855909_24.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/0f1443a5-950b-412e-b7ad-aa490d0a0898_isrForCrosstalkSources_855909_24.stderr parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 102 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 102 has memoization hash b4a243ec0286a7403c61067791951dea parsl.dataflow.memoization INFO: Task 102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de041b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 103 parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/22748c6a-cc63-49a3-aaf2-eb7283d034ef_isrForCrosstalkSources_855821_34.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/22748c6a-cc63-49a3-aaf2-eb7283d034ef_isrForCrosstalkSources_855821_34.stderr parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 103 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 103 has memoization hash ae8ce2a1a77e418000fb07f7e7cadc6c parsl.dataflow.memoization INFO: Task 103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de041c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 104 parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/47a06708-b3b2-4f48-bacc-cc36bb60a7c0_isrForCrosstalkSources_855821_58.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/47a06708-b3b2-4f48-bacc-cc36bb60a7c0_isrForCrosstalkSources_855821_58.stderr parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 104 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 104 has memoization hash ab1ac6c105e10e5315a7a6aa4afcbef5 parsl.dataflow.memoization INFO: Task 104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de041bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 105 parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/6511eb27-d0ea-4b76-a030-26342eb1ae4a_isrForCrosstalkSources_855909_20.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/6511eb27-d0ea-4b76-a030-26342eb1ae4a_isrForCrosstalkSources_855909_20.stderr parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 105 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 105 has memoization hash 663ebdd79980a6bf9f6dc31e3371f3aa parsl.dataflow.memoization INFO: Task 105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de041d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 106 parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/a27f3f12-3334-4676-8e45-4a25e4011e44_isrForCrosstalkSources_855822_2.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/a27f3f12-3334-4676-8e45-4a25e4011e44_isrForCrosstalkSources_855822_2.stderr parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 106 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 106 has memoization hash 302448d0ef27cefb1a0be72580bc8d26 parsl.dataflow.memoization INFO: Task 106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de041da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 107 parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/300ffd52-09fe-4c60-bc8c-ffff4986b713_isrForCrosstalkSources_855910_37.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/300ffd52-09fe-4c60-bc8c-ffff4986b713_isrForCrosstalkSources_855910_37.stderr parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 107 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 107 has memoization hash 32bd7f106f861555b90917b168b3da0b parsl.dataflow.memoization INFO: Task 107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de041e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 108 parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/dccb49ac-546a-4d82-a10c-60ed178995d8_isrForCrosstalkSources_855909_15.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/dccb49ac-546a-4d82-a10c-60ed178995d8_isrForCrosstalkSources_855909_15.stderr parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 108 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 108 has memoization hash 407f6c8f49ba3d2628419dd6062a7ff3 parsl.dataflow.memoization INFO: Task 108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de041f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 109 parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/5740cecc-a40c-42b3-8d8b-951f6c97bc76_isrForCrosstalkSources_855821_56.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/5740cecc-a40c-42b3-8d8b-951f6c97bc76_isrForCrosstalkSources_855821_56.stderr parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 109 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 109 has memoization hash e8d9ccd3a8deb03a2188b3fe6e4ac067 parsl.dataflow.memoization INFO: Task 109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de041ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 110 parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/d2f7d4bf-ae71-4be3-bf7c-854e7f9332e4_isrForCrosstalkSources_855911_9.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/d2f7d4bf-ae71-4be3-bf7c-854e7f9332e4_isrForCrosstalkSources_855911_9.stderr parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 110 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 110 has memoization hash 062fde19b3aa7a4225bddcc2cc36b75a parsl.dataflow.memoization INFO: Task 110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de042020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 111 parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/fcc3a775-e860-4609-9a27-506efd9919b1_isrForCrosstalkSources_855910_28.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/fcc3a775-e860-4609-9a27-506efd9919b1_isrForCrosstalkSources_855910_28.stderr parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 111 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 111 has memoization hash e09ae6dccb1b0f70da982773addcaeb8 parsl.dataflow.memoization INFO: Task 111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de0420c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 112 parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/89b3074c-8e17-44e6-b968-829afac8baeb_isrForCrosstalkSources_855910_49.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/89b3074c-8e17-44e6-b968-829afac8baeb_isrForCrosstalkSources_855910_49.stderr parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 112 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 112 has memoization hash 53a718b6d56d56139786b490d8643dc0 parsl.dataflow.memoization INFO: Task 112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de042160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 113 parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/3563190b-efa4-4097-80a2-0a88ed68ed85_isrForCrosstalkSources_855910_39.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/3563190b-efa4-4097-80a2-0a88ed68ed85_isrForCrosstalkSources_855910_39.stderr parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 113 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 113 has memoization hash cdeb689f5dfb653e3b7e7b3e401ea78b parsl.dataflow.memoization INFO: Task 113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de0422a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 114 parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/cbb64a1e-5a75-4779-a08d-87075a7a0651_isrForCrosstalkSources_855820_29.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/cbb64a1e-5a75-4779-a08d-87075a7a0651_isrForCrosstalkSources_855820_29.stderr parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 114 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 114 has memoization hash 2d717c0ee1b45c19ce038dc00f6ffa12 parsl.dataflow.memoization INFO: Task 114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de042200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 115 parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/513f68d2-6d7d-43e3-92be-f4c56d2ffb21_isrForCrosstalkSources_855909_34.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/513f68d2-6d7d-43e3-92be-f4c56d2ffb21_isrForCrosstalkSources_855909_34.stderr parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 115 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 115 has memoization hash 139a94c01137cfe28829aacc6aae4dab parsl.dataflow.memoization INFO: Task 115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de042340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 116 parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/76d9ff32-c0c7-44fb-b014-58f9f4546a5e_isrForCrosstalkSources_855821_12.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/76d9ff32-c0c7-44fb-b014-58f9f4546a5e_isrForCrosstalkSources_855821_12.stderr parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 116 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 116 has memoization hash d914e1ade9feb882344b6dc4de47f9ff parsl.dataflow.memoization INFO: Task 116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de0423e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 117 parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/b95aa468-c53e-4189-b775-a235a57215e0_isrForCrosstalkSources_855911_7.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/b95aa468-c53e-4189-b775-a235a57215e0_isrForCrosstalkSources_855911_7.stderr parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 117 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 117 has memoization hash b3f2f8cc86a254dfb9517c9ccd68b80c parsl.dataflow.memoization INFO: Task 117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de042480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 118 parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/09851928-2ab7-4cbd-b189-5ae91e72dc79_isrForCrosstalkSources_855821_6.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/09851928-2ab7-4cbd-b189-5ae91e72dc79_isrForCrosstalkSources_855821_6.stderr parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 118 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash da1d3246864f79e679fb633991de51fe parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de0425c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 119 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/30ebd418-98b8-4074-b9be-b22fca161ebb_isrForCrosstalkSources_855909_43.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/30ebd418-98b8-4074-b9be-b22fca161ebb_isrForCrosstalkSources_855909_43.stderr parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 119 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 119 has memoization hash c8828440444a6692a2737c7f9369e81f parsl.dataflow.memoization INFO: Task 119 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de042520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 119 try 0 launched on executor multi with executor id 120 parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/ad9ca9de-09a1-4a10-b92e-c75ce37471bf_isrForCrosstalkSources_855909_52.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/ad9ca9de-09a1-4a10-b92e-c75ce37471bf_isrForCrosstalkSources_855909_52.stderr parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 120 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 120 has memoization hash 58f9e74a98d625bc50710534dfb8d697 parsl.dataflow.memoization INFO: Task 120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de042660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 121 parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/6465ec59-9a5f-44f2-b67c-a1e63452b2c2_isrForCrosstalkSources_855910_47.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/6465ec59-9a5f-44f2-b67c-a1e63452b2c2_isrForCrosstalkSources_855910_47.stderr parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 121 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 121 has memoization hash 2d63c8526d60321457c4aa1ed750e900 parsl.dataflow.memoization INFO: Task 121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de042700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 122 parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/3d1340c7-7322-45d9-950b-f731f19ef39a_isrForCrosstalkSources_855910_14.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/3d1340c7-7322-45d9-950b-f731f19ef39a_isrForCrosstalkSources_855910_14.stderr parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 122 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 122 has memoization hash 0bfe2a89c8c6c87a90e41ff8bfe972aa parsl.dataflow.memoization INFO: Task 122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de0427a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 123 parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/983b57d9-d701-4447-919b-fab33cf21818_isrForCrosstalkSources_855821_14.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/983b57d9-d701-4447-919b-fab33cf21818_isrForCrosstalkSources_855821_14.stderr parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 123 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 123 has memoization hash 52c4003cb47012f7a089d81c1f201575 parsl.dataflow.memoization INFO: Task 123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de0428e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 124 parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/447c03b9-153d-4248-a444-ef511e9ca82e_isrForCrosstalkSources_855911_47.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/447c03b9-153d-4248-a444-ef511e9ca82e_isrForCrosstalkSources_855911_47.stderr parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 124 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 124 has memoization hash 752be8fefadf89977c9cbf693256430c parsl.dataflow.memoization INFO: Task 124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de042840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 125 parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/71040246-eed5-48c9-85a9-4a59d7c0c429_isrForCrosstalkSources_855821_31.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/71040246-eed5-48c9-85a9-4a59d7c0c429_isrForCrosstalkSources_855821_31.stderr parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 125 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 125 has memoization hash 897c44d30c101ee91726ba047be3c583 parsl.dataflow.memoization INFO: Task 125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de042980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 125 try 0 launched on executor multi with executor id 126 parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/039bdf69-bf88-4239-904e-fbe538b340c6_isrForCrosstalkSources_855909_47.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/039bdf69-bf88-4239-904e-fbe538b340c6_isrForCrosstalkSources_855909_47.stderr parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 126 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 126 has memoization hash ef7ae0b9601d4f706557385acff0921c parsl.dataflow.memoization INFO: Task 126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de042a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 127 parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/2b85aa75-02eb-4f68-84f2-24c9f73d0745_isrForCrosstalkSources_855822_13.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/2b85aa75-02eb-4f68-84f2-24c9f73d0745_isrForCrosstalkSources_855822_13.stderr parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 127 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 127 has memoization hash b89340b2ced1921aa4570c85b4da9642 parsl.dataflow.memoization INFO: Task 127 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de042ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 127 try 0 launched on executor multi with executor id 128 parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/fc788203-a9fb-4812-9fe3-7b1487be49c6_isrForCrosstalkSources_855909_17.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/fc788203-a9fb-4812-9fe3-7b1487be49c6_isrForCrosstalkSources_855909_17.stderr parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 128 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 128 has memoization hash 18f68e35ebcc1b5329107f2bd4b1a58b parsl.dataflow.memoization INFO: Task 128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de042c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 129 parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/d675f7d8-4aeb-4a98-a053-fbee038635a1_isrForCrosstalkSources_855820_52.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/d675f7d8-4aeb-4a98-a053-fbee038635a1_isrForCrosstalkSources_855820_52.stderr parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 129 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 129 has memoization hash 2ceb43a42e17eedb65129e4369966b3a parsl.dataflow.memoization INFO: Task 129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de042b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 129 try 0 launched on executor multi with executor id 130 parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/e3bf6294-edc7-4775-afc6-c70037a29eb8_isrForCrosstalkSources_855822_32.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/e3bf6294-edc7-4775-afc6-c70037a29eb8_isrForCrosstalkSources_855822_32.stderr parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 130 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 130 has memoization hash e272a2a9f5e6558feab8c7c8212e583e parsl.dataflow.memoization INFO: Task 130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de042ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 131 parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/0381c584-cae5-45e7-97fc-d9840ba3736b_isrForCrosstalkSources_855911_5.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/0381c584-cae5-45e7-97fc-d9840ba3736b_isrForCrosstalkSources_855911_5.stderr parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 131 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 131 has memoization hash b966f570d0eac8c422f043bd5abf351e parsl.dataflow.memoization INFO: Task 131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de042d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 131 try 0 launched on executor multi with executor id 132 parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/845dc36b-07b1-43e1-90d9-928a88029388_isrForCrosstalkSources_855821_9.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/845dc36b-07b1-43e1-90d9-928a88029388_isrForCrosstalkSources_855821_9.stderr parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 132 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 132 has memoization hash 4fb851258d4f5df5ccbcbd3b0b4690d3 parsl.dataflow.memoization INFO: Task 132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de042de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 133 parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/e0cad1e0-5f58-42ed-81f9-abc4c3671b1f_isrForCrosstalkSources_855822_37.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/e0cad1e0-5f58-42ed-81f9-abc4c3671b1f_isrForCrosstalkSources_855822_37.stderr parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 133 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 133 has memoization hash de24fa06bfdd383ad9ec8ce2bd96e8ec parsl.dataflow.memoization INFO: Task 133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de042f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 133 try 0 launched on executor multi with executor id 134 parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/3baf1cf1-455a-46c3-a9d6-bd6eee64cc98_isrForCrosstalkSources_855820_31.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/3baf1cf1-455a-46c3-a9d6-bd6eee64cc98_isrForCrosstalkSources_855820_31.stderr parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 134 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 134 has memoization hash 0f7486bdb803c7c2294b6976c6e25d84 parsl.dataflow.memoization INFO: Task 134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de042e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 135 parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/d316f3cc-7f2a-45ad-bf4a-695401912528_isrForCrosstalkSources_855910_48.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/d316f3cc-7f2a-45ad-bf4a-695401912528_isrForCrosstalkSources_855910_48.stderr parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 135 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 135 has memoization hash 98ae6369c2be5628eaa23bc8d7a0b9b9 parsl.dataflow.memoization INFO: Task 135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de042fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 135 try 0 launched on executor multi with executor id 136 parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/9a19a9b5-58f1-4dca-9195-3485182ee451_isrForCrosstalkSources_855911_37.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/9a19a9b5-58f1-4dca-9195-3485182ee451_isrForCrosstalkSources_855911_37.stderr parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 136 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 136 has memoization hash a687a01bc8a8959d14fcc8d4eda7a76e parsl.dataflow.memoization INFO: Task 136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de043060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 137 parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/51e03e44-e3cf-42b5-8e75-d661d19c250e_isrForCrosstalkSources_855910_11.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/51e03e44-e3cf-42b5-8e75-d661d19c250e_isrForCrosstalkSources_855910_11.stderr parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 137 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 137 has memoization hash a300248e1b10291fa00bf50ae4a19f86 parsl.dataflow.memoization INFO: Task 137 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de043100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 137 try 0 launched on executor multi with executor id 138 parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/5cdb4000-8389-429a-b6f2-0f89617695e9_isrForCrosstalkSources_855822_17.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/5cdb4000-8389-429a-b6f2-0f89617695e9_isrForCrosstalkSources_855822_17.stderr parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 138 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 138 has memoization hash edb1af494666620000aaa94c0c14f592 parsl.dataflow.memoization INFO: Task 138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de043240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 139 parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/fe27fb3e-eea6-49ac-8c39-d2f16feee535_isrForCrosstalkSources_855820_24.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/fe27fb3e-eea6-49ac-8c39-d2f16feee535_isrForCrosstalkSources_855820_24.stderr parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 139 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 139 has memoization hash 37ac4135c3b4efd1a11a137ee7e40c05 parsl.dataflow.memoization INFO: Task 139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de0431a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 139 try 0 launched on executor multi with executor id 140 parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/6b45ddb3-dec1-410a-8fb9-f79a986ea269_isrForCrosstalkSources_855909_4.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/6b45ddb3-dec1-410a-8fb9-f79a986ea269_isrForCrosstalkSources_855909_4.stderr parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 140 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 140 has memoization hash 2cbb3e4a4fe1a690fac57ec5f2a69313 parsl.dataflow.memoization INFO: Task 140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de0432e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 141 parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/30ac3309-3ae6-412b-b4cf-d2deb8f7c973_isrForCrosstalkSources_855911_48.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/30ac3309-3ae6-412b-b4cf-d2deb8f7c973_isrForCrosstalkSources_855911_48.stderr parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 141 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 141 has memoization hash 895f8ef3a45db32c387b72b4ed257713 parsl.dataflow.memoization INFO: Task 141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de043380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 141 try 0 launched on executor multi with executor id 142 parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/6248229d-b28b-458b-8f8f-f46cc42b9667_isrForCrosstalkSources_855909_44.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/6248229d-b28b-458b-8f8f-f46cc42b9667_isrForCrosstalkSources_855909_44.stderr parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 142 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 142 has memoization hash c6186b9751b47e9cf4c32d780d409668 parsl.dataflow.memoization INFO: Task 142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de043420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 143 parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/a4998646-e1f2-44da-a68b-94c33a400be4_isrForCrosstalkSources_855822_20.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/a4998646-e1f2-44da-a68b-94c33a400be4_isrForCrosstalkSources_855822_20.stderr parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 143 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 143 has memoization hash 2a4cdc36f7a362ac35ec545727b979dd parsl.dataflow.memoization INFO: Task 143 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de043560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 143 try 0 launched on executor multi with executor id 144 parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/ceb1f1e8-be09-422f-b984-66ad8211013e_isrForCrosstalkSources_855821_62.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/ceb1f1e8-be09-422f-b984-66ad8211013e_isrForCrosstalkSources_855821_62.stderr parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 144 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 144 has memoization hash 92e408376dc8e304de86ac7607839bfe parsl.dataflow.memoization INFO: Task 144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de0434c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 145 parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/8c85f8ac-c36d-48d2-8b06-2ce0e0065448_isrForCrosstalkSources_855909_9.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/8c85f8ac-c36d-48d2-8b06-2ce0e0065448_isrForCrosstalkSources_855909_9.stderr parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 145 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 145 has memoization hash 383deadd7a707c5cc7fc6a1e0a1ccf2a parsl.dataflow.memoization INFO: Task 145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de043600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 145 try 0 launched on executor multi with executor id 146 parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/5b6b3b77-2cb3-46a4-9f1b-19057856adc9_isrForCrosstalkSources_855911_50.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/5b6b3b77-2cb3-46a4-9f1b-19057856adc9_isrForCrosstalkSources_855911_50.stderr parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 146 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 146 has memoization hash 836ef952510d4dffee6e8bcc03a3a8ee parsl.dataflow.memoization INFO: Task 146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de0436a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 146 try 0 launched on executor multi with executor id 147 parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/e6aaddc9-fd8f-49be-84dc-20c5c4629ce1_isrForCrosstalkSources_855911_38.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/e6aaddc9-fd8f-49be-84dc-20c5c4629ce1_isrForCrosstalkSources_855911_38.stderr parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 147 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 147 has memoization hash a91aa42740b5c162c9dd708b24f51c1d parsl.dataflow.memoization INFO: Task 147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de043740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 147 try 0 launched on executor multi with executor id 148 parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/db3a81ac-34e1-4aad-9cbc-996485944580_isrForCrosstalkSources_855909_51.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/db3a81ac-34e1-4aad-9cbc-996485944580_isrForCrosstalkSources_855909_51.stderr parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 148 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 148 has memoization hash bdea59d309b1c284b9f851e7aaec2149 parsl.dataflow.memoization INFO: Task 148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de043880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 149 parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/c9f2dae1-405e-4f2a-96eb-55d192d731a5_isrForCrosstalkSources_855822_29.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/c9f2dae1-405e-4f2a-96eb-55d192d731a5_isrForCrosstalkSources_855822_29.stderr parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 149 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 149 has memoization hash bb020c78ed846b3baa3ee63f75e3f867 parsl.dataflow.memoization INFO: Task 149 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de0437e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 149 try 0 launched on executor multi with executor id 150 parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/6cf38398-1445-45f5-89c8-da4cf16d7374_isrForCrosstalkSources_855821_16.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/6cf38398-1445-45f5-89c8-da4cf16d7374_isrForCrosstalkSources_855821_16.stderr parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 150 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 150 has memoization hash 3cfc768f6d121d1fbab87e55a3929a8c parsl.dataflow.memoization INFO: Task 150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de043920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 151 parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/d8159da8-653f-4552-8923-08359445607b_isrForCrosstalkSources_855821_52.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/d8159da8-653f-4552-8923-08359445607b_isrForCrosstalkSources_855821_52.stderr parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 151 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 151 has memoization hash c677d87582828103716217667e97c9b1 parsl.dataflow.memoization INFO: Task 151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de0439c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 151 try 0 launched on executor multi with executor id 152 parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/dfca7045-e482-4bd2-8c18-5858d7329c8a_isrForCrosstalkSources_855820_4.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/dfca7045-e482-4bd2-8c18-5858d7329c8a_isrForCrosstalkSources_855820_4.stderr parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 152 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 152 has memoization hash ae402526a23e0bd44a6b6f9147a3d6b9 parsl.dataflow.memoization INFO: Task 152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de043a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 152 try 0 launched on executor multi with executor id 153 parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/534fb3f3-53a7-44a0-8cfd-a989ad29de4c_isrForCrosstalkSources_855910_21.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/534fb3f3-53a7-44a0-8cfd-a989ad29de4c_isrForCrosstalkSources_855910_21.stderr parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 153 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 153 has memoization hash 52cd70407d70a39820fe523e94dffb19 parsl.dataflow.memoization INFO: Task 153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de043ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 153 try 0 launched on executor multi with executor id 154 parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/d430a535-e9aa-4ef0-8017-6da8669b3bac_isrForCrosstalkSources_855909_5.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/d430a535-e9aa-4ef0-8017-6da8669b3bac_isrForCrosstalkSources_855909_5.stderr parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 154 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 154 has memoization hash 52550898a900e6c91f21123670378b7a parsl.dataflow.memoization INFO: Task 154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de043b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 155 parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/8b79b70a-2fb3-414f-aefd-b2b5f6cfb8dd_isrForCrosstalkSources_855910_38.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/8b79b70a-2fb3-414f-aefd-b2b5f6cfb8dd_isrForCrosstalkSources_855910_38.stderr parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 155 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 155 has memoization hash 0e2982b176ffa42950b41c2c1e5e8643 parsl.dataflow.memoization INFO: Task 155 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de043c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 155 try 0 launched on executor multi with executor id 156 parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/0e1ab3f7-ebe2-46d5-8169-4fc9d83e5ccf_isrForCrosstalkSources_855910_53.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/0e1ab3f7-ebe2-46d5-8169-4fc9d83e5ccf_isrForCrosstalkSources_855910_53.stderr parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 156 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 156 has memoization hash 2aa4c1b434eb62c105ece206e0dfeaf4 parsl.dataflow.memoization INFO: Task 156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de043ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 157 parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/a147c4a9-9115-4a68-8536-68f1b82c7011_isrForCrosstalkSources_855822_45.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/a147c4a9-9115-4a68-8536-68f1b82c7011_isrForCrosstalkSources_855822_45.stderr parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 157 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 157 has memoization hash 1742e6ca80ff07050ab9a70eae84cd13 parsl.dataflow.memoization INFO: Task 157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de043d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 157 try 0 launched on executor multi with executor id 158 parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/c92885f0-3f33-49e2-8de5-e7c643078c3e_isrForCrosstalkSources_855821_18.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/c92885f0-3f33-49e2-8de5-e7c643078c3e_isrForCrosstalkSources_855821_18.stderr parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 158 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 158 has memoization hash bd7f9c7776857a571d19a1f250850fd0 parsl.dataflow.memoization INFO: Task 158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de043ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 158 try 0 launched on executor multi with executor id 159 parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/5f695f4b-33ba-40ad-87e4-30cf51cc961b_isrForCrosstalkSources_855909_28.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/5f695f4b-33ba-40ad-87e4-30cf51cc961b_isrForCrosstalkSources_855909_28.stderr parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 159 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 159 has memoization hash fcfbe7ebd37346e6c10bad29640ef4f0 parsl.dataflow.memoization INFO: Task 159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de043e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 159 try 0 launched on executor multi with executor id 160 parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/60f679ef-3b94-4266-b907-149f5c577053_isrForCrosstalkSources_855911_34.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/60f679ef-3b94-4266-b907-149f5c577053_isrForCrosstalkSources_855911_34.stderr parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 160 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 160 has memoization hash 2449d97553feaadce03bb20627cf67c2 parsl.dataflow.memoization INFO: Task 160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de043f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 160 try 0 launched on executor multi with executor id 161 parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/f3089494-b88b-45e6-9045-651586aa7c32_isrForCrosstalkSources_855911_30.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/f3089494-b88b-45e6-9045-651586aa7c32_isrForCrosstalkSources_855911_30.stderr parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 161 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 161 has memoization hash 66053d82bd47cb65a6d475d2a85941ff parsl.dataflow.memoization INFO: Task 161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3de040040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 161 try 0 launched on executor multi with executor id 162 parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/c41cfc8e-fc60-48c6-a8c2-f31e1760dd40_isrForCrosstalkSources_855821_53.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/c41cfc8e-fc60-48c6-a8c2-f31e1760dd40_isrForCrosstalkSources_855821_53.stderr parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 162 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 162 has memoization hash 1de4cb593fb57d7fcddfc51582357d43 parsl.dataflow.memoization INFO: Task 162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca840e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 163 parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/46751165-d85d-48c6-bcff-fd91a79a0d61_isrForCrosstalkSources_855911_1.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/46751165-d85d-48c6-bcff-fd91a79a0d61_isrForCrosstalkSources_855911_1.stderr parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 163 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 163 has memoization hash 7a5825609c3c2d4827ad69aaec431e9b parsl.dataflow.memoization INFO: Task 163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca84220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 163 try 0 launched on executor multi with executor id 164 parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/1723b53a-e927-40ff-9c0b-77ba8c663bac_isrForCrosstalkSources_855821_3.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/1723b53a-e927-40ff-9c0b-77ba8c663bac_isrForCrosstalkSources_855821_3.stderr parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 164 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 164 has memoization hash 8f376f7965280759bec1610e25b7847f parsl.dataflow.memoization INFO: Task 164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca84180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 165 parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/e576b03a-2a40-4b2b-a242-6434db2a7219_isrForCrosstalkSources_855822_4.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/e576b03a-2a40-4b2b-a242-6434db2a7219_isrForCrosstalkSources_855822_4.stderr parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 165 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 165 has memoization hash 34f2b16acf0c6055b6f6f60875f08844 parsl.dataflow.memoization INFO: Task 165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca842c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 165 try 0 launched on executor multi with executor id 166 parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/91c9eab0-d671-4271-86bd-64743e9b600b_isrForCrosstalkSources_855820_22.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/91c9eab0-d671-4271-86bd-64743e9b600b_isrForCrosstalkSources_855820_22.stderr parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 166 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 166 has memoization hash 04f83bbcab22385dcaa07d021425faf3 parsl.dataflow.memoization INFO: Task 166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca84360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 167 parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/4980235f-c3b9-4dd4-a46d-3a2207661509_isrForCrosstalkSources_855909_26.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/4980235f-c3b9-4dd4-a46d-3a2207661509_isrForCrosstalkSources_855909_26.stderr parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 167 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 167 has memoization hash 5fb04d1e92ad5e6e083a0c89f0ed911b parsl.dataflow.memoization INFO: Task 167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca84400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 167 try 0 launched on executor multi with executor id 168 parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/c8a77bb7-5b75-40d5-a0c3-3c1b5f659ce4_isrForCrosstalkSources_855821_2.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/c8a77bb7-5b75-40d5-a0c3-3c1b5f659ce4_isrForCrosstalkSources_855821_2.stderr parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 168 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 168 has memoization hash 51be36d3c7057a8f9426e1fe7d8a2e54 parsl.dataflow.memoization INFO: Task 168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca84540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 169 parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/d91531a8-c97c-4ed6-8765-fa35798ee4db_isrForCrosstalkSources_855911_24.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/d91531a8-c97c-4ed6-8765-fa35798ee4db_isrForCrosstalkSources_855911_24.stderr parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 169 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 169 has memoization hash c249ca2b518a76967dfe694a31a68177 parsl.dataflow.memoization INFO: Task 169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca844a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 169 try 0 launched on executor multi with executor id 170 parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/8ae7e14a-6c23-413a-93f7-8b185bee19d6_isrForCrosstalkSources_855909_18.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/8ae7e14a-6c23-413a-93f7-8b185bee19d6_isrForCrosstalkSources_855909_18.stderr parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 170 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 170 has memoization hash 9fcd39cd035845ad9f614a7a09f32d9d parsl.dataflow.memoization INFO: Task 170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca845e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 170 try 0 launched on executor multi with executor id 171 parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/7148c205-e875-404c-a644-661b6acda94f_isrForCrosstalkSources_855821_17.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/7148c205-e875-404c-a644-661b6acda94f_isrForCrosstalkSources_855821_17.stderr parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 171 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 171 has memoization hash 4ae91de8f379bb756d482c35f9e47d7f parsl.dataflow.memoization INFO: Task 171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca84680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 171 try 0 launched on executor multi with executor id 172 parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/081d6598-d26e-47f7-a585-19d5bf276c17_isrForCrosstalkSources_855820_37.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/081d6598-d26e-47f7-a585-19d5bf276c17_isrForCrosstalkSources_855820_37.stderr parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 172 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 172 has memoization hash ca4b338c55dca5747e3fa69d0ea95c20 parsl.dataflow.memoization INFO: Task 172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca84720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 173 parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/3ada3f31-68ef-4ed8-9f82-53d373c2b504_isrForCrosstalkSources_855910_33.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/3ada3f31-68ef-4ed8-9f82-53d373c2b504_isrForCrosstalkSources_855910_33.stderr parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 173 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 173 has memoization hash 3afaf031ec53a4fb7107d49e3d45b36e parsl.dataflow.memoization INFO: Task 173 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca84860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 173 try 0 launched on executor multi with executor id 174 parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/182a5abe-5aaf-4534-9354-2468bfc97a30_isrForCrosstalkSources_855910_6.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/182a5abe-5aaf-4534-9354-2468bfc97a30_isrForCrosstalkSources_855910_6.stderr parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 174 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 174 has memoization hash 7ad6ecb273afd22792c209ddf7437de5 parsl.dataflow.memoization INFO: Task 174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca847c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 175 parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/14329ff8-9520-4ac2-9e79-b14b2862d5c3_isrForCrosstalkSources_855821_27.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/14329ff8-9520-4ac2-9e79-b14b2862d5c3_isrForCrosstalkSources_855821_27.stderr parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 175 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 175 has memoization hash 941441e10bd0737336fc4596ca88443d parsl.dataflow.memoization INFO: Task 175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca84900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 175 try 0 launched on executor multi with executor id 176 parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/9366051c-cb15-4d4c-9d6f-05b84dae8156_isrForCrosstalkSources_855820_27.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/9366051c-cb15-4d4c-9d6f-05b84dae8156_isrForCrosstalkSources_855820_27.stderr parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 176 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 176 has memoization hash 03bb741e7cd973ac7e6abdc90de42da3 parsl.dataflow.memoization INFO: Task 176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca849a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 177 parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/56187f5b-d739-4f75-87a4-645cbc9a1827_isrForCrosstalkSources_855909_41.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/56187f5b-d739-4f75-87a4-645cbc9a1827_isrForCrosstalkSources_855909_41.stderr parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 177 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 177 has memoization hash 491b867208d284c56c71ebed2c59a8c6 parsl.dataflow.memoization INFO: Task 177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca84a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 177 try 0 launched on executor multi with executor id 178 parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/7d178d57-5c50-4060-b06b-2c606e49e0cd_isrForCrosstalkSources_855909_61.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/7d178d57-5c50-4060-b06b-2c606e49e0cd_isrForCrosstalkSources_855909_61.stderr parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 178 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 178 has memoization hash 09c9daf3694aa212f35dd552558372cd parsl.dataflow.memoization INFO: Task 178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca84b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 179 parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/12191924-91e5-473f-b886-4eabc8e03a06_isrForCrosstalkSources_855821_4.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/12191924-91e5-473f-b886-4eabc8e03a06_isrForCrosstalkSources_855821_4.stderr parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 179 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 179 has memoization hash d64e8e2f409ab89d2ed89c0e6369c420 parsl.dataflow.memoization INFO: Task 179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca84ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 179 try 0 launched on executor multi with executor id 180 parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/a70a3d07-573b-41a3-b9d9-2e0957730ec1_isrForCrosstalkSources_855820_3.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/a70a3d07-573b-41a3-b9d9-2e0957730ec1_isrForCrosstalkSources_855820_3.stderr parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 180 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 180 has memoization hash 4a07742b5eeacb574e52064b32eb637f parsl.dataflow.memoization INFO: Task 180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca84c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 181 parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/8344e640-81ad-47ea-b218-99018068f49b_isrForCrosstalkSources_855822_36.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/8344e640-81ad-47ea-b218-99018068f49b_isrForCrosstalkSources_855822_36.stderr parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 181 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 181 has memoization hash 18f4ae4f33a677f248e3652ca0f444c8 parsl.dataflow.memoization INFO: Task 181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca84cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 181 try 0 launched on executor multi with executor id 182 parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/349659b3-3964-40bd-9891-8ea26fe9f618_isrForCrosstalkSources_855820_61.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/349659b3-3964-40bd-9891-8ea26fe9f618_isrForCrosstalkSources_855820_61.stderr parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 182 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 182 has memoization hash be62cc9f7e541094b9a111ea358d32f1 parsl.dataflow.memoization INFO: Task 182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca84d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 183 parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/2553a92b-9da3-44fe-ae33-b4819965fc17_isrForCrosstalkSources_855821_39.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/2553a92b-9da3-44fe-ae33-b4819965fc17_isrForCrosstalkSources_855821_39.stderr parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 183 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 183 has memoization hash 6d980291df15588c91befbd8dc114778 parsl.dataflow.memoization INFO: Task 183 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca84ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 183 try 0 launched on executor multi with executor id 184 parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/26e3cb7b-388f-460a-9962-06877df2a3bd_isrForCrosstalkSources_855821_47.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/26e3cb7b-388f-460a-9962-06877df2a3bd_isrForCrosstalkSources_855821_47.stderr parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 184 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 184 has memoization hash 6f05423ad6766143356a50969a56140d parsl.dataflow.memoization INFO: Task 184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca84e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 185 parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/233d4275-25ae-47d9-8c00-07493a2e3548_isrForCrosstalkSources_855821_48.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/233d4275-25ae-47d9-8c00-07493a2e3548_isrForCrosstalkSources_855821_48.stderr parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 185 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 185 has memoization hash 44bf0f021a7ef8d79891ee81d4c1aad4 parsl.dataflow.memoization INFO: Task 185 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca84f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 185 try 0 launched on executor multi with executor id 186 parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/6d58b8c2-cdd5-4242-abd7-9a16527f0a00_isrForCrosstalkSources_855822_62.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/6d58b8c2-cdd5-4242-abd7-9a16527f0a00_isrForCrosstalkSources_855822_62.stderr parsl.dataflow.dflow DEBUG: Task 186 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 186 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 186 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 186 has memoization hash 4dfa77ce34de3d2cf77e217a7baf1826 parsl.dataflow.memoization INFO: Task 186 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca84fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 186 try 0 launched on executor multi with executor id 187 parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/2a9b5761-40a2-42ca-b299-83b98e3aedd8_isrForCrosstalkSources_855821_23.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/2a9b5761-40a2-42ca-b299-83b98e3aedd8_isrForCrosstalkSources_855821_23.stderr parsl.dataflow.dflow DEBUG: Task 187 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 187 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 187 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 187 has memoization hash 04edbe5cb8ab6b0223981f2fd1ba08b4 parsl.dataflow.memoization INFO: Task 187 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca85080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 187 try 0 launched on executor multi with executor id 188 parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/056c8cf9-32f8-465e-9e1b-80d5287e8943_isrForCrosstalkSources_855821_21.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/056c8cf9-32f8-465e-9e1b-80d5287e8943_isrForCrosstalkSources_855821_21.stderr parsl.dataflow.dflow DEBUG: Task 188 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 188 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 188 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 188 has memoization hash 02b5619dfe77e079ff11c5c83f0a3967 parsl.dataflow.memoization INFO: Task 188 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca851c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 188 try 0 launched on executor multi with executor id 189 parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/1ed07551-3258-44aa-aa51-2cc1ab0146dd_isrForCrosstalkSources_855910_51.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/1ed07551-3258-44aa-aa51-2cc1ab0146dd_isrForCrosstalkSources_855910_51.stderr parsl.dataflow.dflow DEBUG: Task 189 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 189 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 189 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 189 has memoization hash 59a6b0686e764a1937e866011ad011f8 parsl.dataflow.memoization INFO: Task 189 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca85120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 189 try 0 launched on executor multi with executor id 190 parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/d8bb0496-f0e9-43b4-a194-5c7e5d222660_isrForCrosstalkSources_855911_18.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/d8bb0496-f0e9-43b4-a194-5c7e5d222660_isrForCrosstalkSources_855911_18.stderr parsl.dataflow.dflow DEBUG: Task 190 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 190 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 190 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 190 has memoization hash 646ac596b6ad1f99fdcad5fde3ba20ab parsl.dataflow.memoization INFO: Task 190 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca85260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 190 try 0 launched on executor multi with executor id 191 parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/87fd7787-c531-4204-ae51-131762f53bf5_isrForCrosstalkSources_855911_36.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/87fd7787-c531-4204-ae51-131762f53bf5_isrForCrosstalkSources_855911_36.stderr parsl.dataflow.dflow DEBUG: Task 191 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 191 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 191 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 191 has memoization hash 809efbad1121da079deff44f03e2d4ad parsl.dataflow.memoization INFO: Task 191 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca85300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 191 try 0 launched on executor multi with executor id 192 parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/5601c7a6-39e4-462f-8ac0-95ebb3809aaf_isrForCrosstalkSources_855820_36.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/5601c7a6-39e4-462f-8ac0-95ebb3809aaf_isrForCrosstalkSources_855820_36.stderr parsl.dataflow.dflow DEBUG: Task 192 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 192 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 192 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 192 has memoization hash c8b6d4d9b535902460d2133214c3fae2 parsl.dataflow.memoization INFO: Task 192 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca853a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 192 try 0 launched on executor multi with executor id 193 parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/29f7e352-7baa-4b0c-92f5-efd11679d3da_isrForCrosstalkSources_855820_13.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/29f7e352-7baa-4b0c-92f5-efd11679d3da_isrForCrosstalkSources_855820_13.stderr parsl.dataflow.dflow DEBUG: Task 193 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 193 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 193 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 193 has memoization hash 234623ad90897556b1c4d7d9c443deab parsl.dataflow.memoization INFO: Task 193 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca854e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 193 try 0 launched on executor multi with executor id 194 parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/2ad53253-bb4d-457c-99b9-26f013fb105b_isrForCrosstalkSources_855822_22.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/2ad53253-bb4d-457c-99b9-26f013fb105b_isrForCrosstalkSources_855822_22.stderr parsl.dataflow.dflow DEBUG: Task 194 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 194 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 194 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 194 has memoization hash aaf8ab5d7e2b14049a4650bab33674f3 parsl.dataflow.memoization INFO: Task 194 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca85440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 194 try 0 launched on executor multi with executor id 195 parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/1b936b99-7e9a-4b76-a8f2-3b86cdce5093_isrForCrosstalkSources_855909_56.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/1b936b99-7e9a-4b76-a8f2-3b86cdce5093_isrForCrosstalkSources_855909_56.stderr parsl.dataflow.dflow DEBUG: Task 195 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 195 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 195 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 195 has memoization hash 3fb1091897ee2fef69d81ba0e13347e0 parsl.dataflow.memoization INFO: Task 195 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca85580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 195 try 0 launched on executor multi with executor id 196 parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/60e9ad41-e577-46a3-bd15-22b54b9f2725_isrForCrosstalkSources_855910_25.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/60e9ad41-e577-46a3-bd15-22b54b9f2725_isrForCrosstalkSources_855910_25.stderr parsl.dataflow.dflow DEBUG: Task 196 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 196 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 196 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 196 has memoization hash a8f7f3b1a31b2e903e76c2d715c276fb parsl.dataflow.memoization INFO: Task 196 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca85620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 196 try 0 launched on executor multi with executor id 197 parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/fcc359f6-5bf4-4953-9ae3-c3ecb9409981_isrForCrosstalkSources_855909_49.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/fcc359f6-5bf4-4953-9ae3-c3ecb9409981_isrForCrosstalkSources_855909_49.stderr parsl.dataflow.dflow DEBUG: Task 197 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 197 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 197 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 197 has memoization hash b125b58ac9e57d56cfbb4b268dbae37f parsl.dataflow.memoization INFO: Task 197 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca856c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 197 try 0 launched on executor multi with executor id 198 parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/153c2c9a-2ba6-42a9-8757-357cc21aff11_isrForCrosstalkSources_855911_28.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/153c2c9a-2ba6-42a9-8757-357cc21aff11_isrForCrosstalkSources_855911_28.stderr parsl.dataflow.dflow DEBUG: Task 198 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 198 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 198 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 198 has memoization hash e2a3bca4fba7e5f8a8a8314a474bcc60 parsl.dataflow.memoization INFO: Task 198 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca85800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 198 try 0 launched on executor multi with executor id 199 parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/ca4fae14-b203-47a0-a6d5-7662d2ccf733_isrForCrosstalkSources_855910_35.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/ca4fae14-b203-47a0-a6d5-7662d2ccf733_isrForCrosstalkSources_855910_35.stderr parsl.dataflow.dflow DEBUG: Task 199 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 199 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 199 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 199 has memoization hash d64253bb872811f9a09d7faad408b212 parsl.dataflow.memoization INFO: Task 199 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca85760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 199 try 0 launched on executor multi with executor id 200 parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/4c2a492e-a8c4-4ffa-9715-e68ab00f4490_isrForCrosstalkSources_855909_6.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/4c2a492e-a8c4-4ffa-9715-e68ab00f4490_isrForCrosstalkSources_855909_6.stderr parsl.dataflow.dflow DEBUG: Task 200 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 200 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 200 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 200 has memoization hash a12fede80d5b2ab9bda24395ba858ff5 parsl.dataflow.memoization INFO: Task 200 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca858a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 200 try 0 launched on executor multi with executor id 201 parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/13ace5da-2534-436b-98a4-04467a6e5116_isrForCrosstalkSources_855822_34.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/13ace5da-2534-436b-98a4-04467a6e5116_isrForCrosstalkSources_855822_34.stderr parsl.dataflow.dflow DEBUG: Task 201 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 201 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 201 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 201 has memoization hash 44f5c7a7014171583d6de3b64efc7dd8 parsl.dataflow.memoization INFO: Task 201 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca85940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 201 try 0 launched on executor multi with executor id 202 parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/3e3484a8-8c02-4365-9bee-c5794880e13b_isrForCrosstalkSources_855820_1.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/3e3484a8-8c02-4365-9bee-c5794880e13b_isrForCrosstalkSources_855820_1.stderr parsl.dataflow.dflow DEBUG: Task 202 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 202 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 202 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 202 has memoization hash a2bdceb1dc15a106fd6ec58179161f47 parsl.dataflow.memoization INFO: Task 202 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca859e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 202 try 0 launched on executor multi with executor id 203 parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/397e2271-e1a6-45d2-9b8f-3b18d29daddc_isrForCrosstalkSources_855911_25.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/397e2271-e1a6-45d2-9b8f-3b18d29daddc_isrForCrosstalkSources_855911_25.stderr parsl.dataflow.dflow DEBUG: Task 203 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 203 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 203 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 203 has memoization hash 146132281fd051bf44a0a25338179f9f parsl.dataflow.memoization INFO: Task 203 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca85b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 203 try 0 launched on executor multi with executor id 204 parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/3ebcb130-cd45-4ebb-acfc-0c4f486b8e93_isrForCrosstalkSources_855910_40.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/3ebcb130-cd45-4ebb-acfc-0c4f486b8e93_isrForCrosstalkSources_855910_40.stderr parsl.dataflow.dflow DEBUG: Task 204 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 204 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 204 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 204 has memoization hash 7320ee6031ac476d890abbd44a4726f0 parsl.dataflow.memoization INFO: Task 204 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca85a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 204 try 0 launched on executor multi with executor id 205 parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/32586c7f-741d-4c97-9b10-a4868dcf77b8_isrForCrosstalkSources_855911_12.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/32586c7f-741d-4c97-9b10-a4868dcf77b8_isrForCrosstalkSources_855911_12.stderr parsl.dataflow.dflow DEBUG: Task 205 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 205 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 205 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 205 has memoization hash 00802b939b4b836d41a1a38cbac31e53 parsl.dataflow.memoization INFO: Task 205 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca85bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 205 try 0 launched on executor multi with executor id 206 parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/6681672f-3839-459e-8d88-bc7d348aaf72_isrForCrosstalkSources_855909_39.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/6681672f-3839-459e-8d88-bc7d348aaf72_isrForCrosstalkSources_855909_39.stderr parsl.dataflow.dflow DEBUG: Task 206 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 206 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 206 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 206 has memoization hash 7f8fa5c409cd046b8a006698041199d3 parsl.dataflow.memoization INFO: Task 206 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca85c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 206 try 0 launched on executor multi with executor id 207 parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/6280b9d8-5ad4-418e-91fe-de72f82997f8_isrForCrosstalkSources_855909_23.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/6280b9d8-5ad4-418e-91fe-de72f82997f8_isrForCrosstalkSources_855909_23.stderr parsl.dataflow.dflow DEBUG: Task 207 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 207 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 207 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 207 has memoization hash 5b9cbb50a64d82461db40787895cc5fb parsl.dataflow.memoization INFO: Task 207 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca85d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 207 try 0 launched on executor multi with executor id 208 parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/5fdee1cb-7588-423f-9321-33dfb530e275_isrForCrosstalkSources_855911_62.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/5fdee1cb-7588-423f-9321-33dfb530e275_isrForCrosstalkSources_855911_62.stderr parsl.dataflow.dflow DEBUG: Task 208 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 208 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 208 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 208 has memoization hash 15cb9428402118666d4e4c1ff606d7b1 parsl.dataflow.memoization INFO: Task 208 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca85e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 208 try 0 launched on executor multi with executor id 209 parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/df91e2f6-fca8-47a6-8a45-faef366e2720_isrForCrosstalkSources_855820_46.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/df91e2f6-fca8-47a6-8a45-faef366e2720_isrForCrosstalkSources_855820_46.stderr parsl.dataflow.dflow DEBUG: Task 209 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 209 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 209 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 209 has memoization hash c6859c838ae3fa0619817ed1f2a8d336 parsl.dataflow.memoization INFO: Task 209 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca85da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 209 try 0 launched on executor multi with executor id 210 parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/a98c121a-be59-4a36-8ecd-8abff1739aae_isrForCrosstalkSources_855910_20.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/a98c121a-be59-4a36-8ecd-8abff1739aae_isrForCrosstalkSources_855910_20.stderr parsl.dataflow.dflow DEBUG: Task 210 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 210 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 210 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 210 has memoization hash adbcd830dcc2efaa31223beabab3002f parsl.dataflow.memoization INFO: Task 210 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca85ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 210 try 0 launched on executor multi with executor id 211 parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/dd8c6e99-39ba-4c67-9932-19951b10011e_isrForCrosstalkSources_855822_8.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/dd8c6e99-39ba-4c67-9932-19951b10011e_isrForCrosstalkSources_855822_8.stderr parsl.dataflow.dflow DEBUG: Task 211 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 211 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 211 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 211 has memoization hash 0ae855a17ce2bbae663e98124efd25e0 parsl.dataflow.memoization INFO: Task 211 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca85f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 211 try 0 launched on executor multi with executor id 212 parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/a302223e-e22e-4972-9905-ac51244fd8dc_isrForCrosstalkSources_855910_4.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/a302223e-e22e-4972-9905-ac51244fd8dc_isrForCrosstalkSources_855910_4.stderr parsl.dataflow.dflow DEBUG: Task 212 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 212 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 212 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 212 has memoization hash a57db97d0fb7ed3ad9586e36dfa42194 parsl.dataflow.memoization INFO: Task 212 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca86020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 212 try 0 launched on executor multi with executor id 213 parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/6098c0e0-4e62-4e68-82fe-ae76d834013f_isrForCrosstalkSources_855909_1.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/6098c0e0-4e62-4e68-82fe-ae76d834013f_isrForCrosstalkSources_855909_1.stderr parsl.dataflow.dflow DEBUG: Task 213 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 213 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 213 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 213 has memoization hash 18c2312cba1096adfb88a30db8eaa1c1 parsl.dataflow.memoization INFO: Task 213 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca86160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 213 try 0 launched on executor multi with executor id 214 parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/0110a7fb-4d00-4a53-87b7-5de31374ca04_isrForCrosstalkSources_855909_46.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/0110a7fb-4d00-4a53-87b7-5de31374ca04_isrForCrosstalkSources_855909_46.stderr parsl.dataflow.dflow DEBUG: Task 214 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 214 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 214 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 214 has memoization hash 3b8c147d7441f34f025bb282435ee0d4 parsl.dataflow.memoization INFO: Task 214 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca860c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 214 try 0 launched on executor multi with executor id 215 parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/3a39aaaa-54cb-4195-a393-fcd691fe6fe7_isrForCrosstalkSources_855910_12.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/3a39aaaa-54cb-4195-a393-fcd691fe6fe7_isrForCrosstalkSources_855910_12.stderr parsl.dataflow.dflow DEBUG: Task 215 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 215 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 215 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 215 has memoization hash 2e4b80fffc9910fa102e4f3ccfb9199e parsl.dataflow.memoization INFO: Task 215 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca86200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 215 try 0 launched on executor multi with executor id 216 parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/c087b1d8-58d0-489a-97ee-0f11048667a0_isrForCrosstalkSources_855820_50.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/c087b1d8-58d0-489a-97ee-0f11048667a0_isrForCrosstalkSources_855820_50.stderr parsl.dataflow.dflow DEBUG: Task 216 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 216 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 216 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 216 has memoization hash 62578a5c2ce295ad79b8838665b24e64 parsl.dataflow.memoization INFO: Task 216 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca862a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 216 try 0 launched on executor multi with executor id 217 parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/0e7b35a7-ede7-49aa-b802-ff407484ff4b_isrForCrosstalkSources_855820_2.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/0e7b35a7-ede7-49aa-b802-ff407484ff4b_isrForCrosstalkSources_855820_2.stderr parsl.dataflow.dflow DEBUG: Task 217 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 217 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 217 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 217 has memoization hash ca18196e44370c7ffb2100b33be86402 parsl.dataflow.memoization INFO: Task 217 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca86340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 217 try 0 launched on executor multi with executor id 218 parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/77c75561-2c65-4ae7-822d-aff25e4b8d4d_isrForCrosstalkSources_855821_59.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/77c75561-2c65-4ae7-822d-aff25e4b8d4d_isrForCrosstalkSources_855821_59.stderr parsl.dataflow.dflow DEBUG: Task 218 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 218 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 218 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 218 has memoization hash c16f7d2a9f99a49e128af80d47336773 parsl.dataflow.memoization INFO: Task 218 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca86480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 218 try 0 launched on executor multi with executor id 219 parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/9f7fde05-8983-45f2-9379-7436cf8d050a_isrForCrosstalkSources_855909_19.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/9f7fde05-8983-45f2-9379-7436cf8d050a_isrForCrosstalkSources_855909_19.stderr parsl.dataflow.dflow DEBUG: Task 219 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 219 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 219 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 219 has memoization hash 82e7263418100a2c829c344ab69872be parsl.dataflow.memoization INFO: Task 219 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca863e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 219 try 0 launched on executor multi with executor id 220 parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/acc25d0b-d59b-44b2-9bd3-704e48066f0f_isrForCrosstalkSources_855820_28.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/acc25d0b-d59b-44b2-9bd3-704e48066f0f_isrForCrosstalkSources_855820_28.stderr parsl.dataflow.dflow DEBUG: Task 220 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 220 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 220 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 220 has memoization hash 43aef52b508aded26fdc2eefbf0908cb parsl.dataflow.memoization INFO: Task 220 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca86520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 220 try 0 launched on executor multi with executor id 221 parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/667d7035-29dc-4073-9c93-7a418399912c_isrForCrosstalkSources_855822_40.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/667d7035-29dc-4073-9c93-7a418399912c_isrForCrosstalkSources_855822_40.stderr parsl.dataflow.dflow DEBUG: Task 221 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 221 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 221 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 221 has memoization hash 0cf52b792cef1f56c7166ee5fa7c481a parsl.dataflow.memoization INFO: Task 221 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca865c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 221 try 0 launched on executor multi with executor id 222 parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/12cc124e-5722-4f7f-9fab-4615dec0c8e1_isrForCrosstalkSources_855911_53.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/12cc124e-5722-4f7f-9fab-4615dec0c8e1_isrForCrosstalkSources_855911_53.stderr parsl.dataflow.dflow DEBUG: Task 222 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 222 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 222 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 222 has memoization hash 1d4a69e71af489c165f166c081279112 parsl.dataflow.memoization INFO: Task 222 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca86660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 222 try 0 launched on executor multi with executor id 223 parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/ece1d310-d293-4b16-bbb4-324cb01623f3_isrForCrosstalkSources_855909_12.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/ece1d310-d293-4b16-bbb4-324cb01623f3_isrForCrosstalkSources_855909_12.stderr parsl.dataflow.dflow DEBUG: Task 223 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 223 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 223 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 223 has memoization hash 85d823e9eac509934403fa2cf73cc89d parsl.dataflow.memoization INFO: Task 223 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca867a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 223 try 0 launched on executor multi with executor id 224 parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/68a2873b-0e09-4bfb-819b-59cd145804f1_isrForCrosstalkSources_855910_55.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/68a2873b-0e09-4bfb-819b-59cd145804f1_isrForCrosstalkSources_855910_55.stderr parsl.dataflow.dflow DEBUG: Task 224 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 224 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 224 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 224 has memoization hash 1292f530adfa3ce9a8138149b791653b parsl.dataflow.memoization INFO: Task 224 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca86700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 224 try 0 launched on executor multi with executor id 225 parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/1ffb075b-fed2-4add-ba8c-86c287059dfc_isrForCrosstalkSources_855910_7.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/1ffb075b-fed2-4add-ba8c-86c287059dfc_isrForCrosstalkSources_855910_7.stderr parsl.dataflow.dflow DEBUG: Task 225 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 225 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 225 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 225 has memoization hash 188653b6103c19db4a483444273cc661 parsl.dataflow.memoization INFO: Task 225 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca86840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 225 try 0 launched on executor multi with executor id 226 parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/ed477725-8c22-424d-9f6c-8b377a2e3a53_isrForCrosstalkSources_855910_52.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/ed477725-8c22-424d-9f6c-8b377a2e3a53_isrForCrosstalkSources_855910_52.stderr parsl.dataflow.dflow DEBUG: Task 226 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 226 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 226 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 226 has memoization hash 5905dddd1e5051311ca2056aed03a82a parsl.dataflow.memoization INFO: Task 226 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca868e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 226 try 0 launched on executor multi with executor id 227 parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/84cc07c0-550b-472c-8306-6f140be05b77_isrForCrosstalkSources_855821_44.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/84cc07c0-550b-472c-8306-6f140be05b77_isrForCrosstalkSources_855821_44.stderr parsl.dataflow.dflow DEBUG: Task 227 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 227 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 227 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 227 has memoization hash e5cdc39a0c14300834da9896205a874c parsl.dataflow.memoization INFO: Task 227 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca86980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 227 try 0 launched on executor multi with executor id 228 parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/466a83ba-9ea3-4fc6-9b47-4adf479126b5_isrForCrosstalkSources_855909_22.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/466a83ba-9ea3-4fc6-9b47-4adf479126b5_isrForCrosstalkSources_855909_22.stderr parsl.dataflow.dflow DEBUG: Task 228 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 228 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 228 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 228 has memoization hash 11e5165930cbccbfe4ff94bbce9c4950 parsl.dataflow.memoization INFO: Task 228 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca86ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 228 try 0 launched on executor multi with executor id 229 parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/9c48a5ef-f7fd-4e6f-8267-3f8cca701161_isrForCrosstalkSources_855909_21.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/9c48a5ef-f7fd-4e6f-8267-3f8cca701161_isrForCrosstalkSources_855909_21.stderr parsl.dataflow.dflow DEBUG: Task 229 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 229 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 229 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 229 has memoization hash baf8f5da895a959df4fd65088c0b4941 parsl.dataflow.memoization INFO: Task 229 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca86a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 229 try 0 launched on executor multi with executor id 230 parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/e83bf6f8-e63a-4308-bb6e-1cf961507c90_isrForCrosstalkSources_855821_30.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/e83bf6f8-e63a-4308-bb6e-1cf961507c90_isrForCrosstalkSources_855821_30.stderr parsl.dataflow.dflow DEBUG: Task 230 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 230 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 230 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 230 has memoization hash 1c6c5091446c2596b11a654212609d35 parsl.dataflow.memoization INFO: Task 230 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca86b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 230 try 0 launched on executor multi with executor id 231 parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/ba7587a0-701d-4dda-be3d-bff46634ce11_isrForCrosstalkSources_855910_36.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/ba7587a0-701d-4dda-be3d-bff46634ce11_isrForCrosstalkSources_855910_36.stderr parsl.dataflow.dflow DEBUG: Task 231 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 231 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 231 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 231 has memoization hash c2f14720c5c3f2b09cbac03cb535e45b parsl.dataflow.memoization INFO: Task 231 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca86c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 231 try 0 launched on executor multi with executor id 232 parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/9c0168d8-aba6-4cfe-bdfd-8c19785da501_isrForCrosstalkSources_855821_55.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/9c0168d8-aba6-4cfe-bdfd-8c19785da501_isrForCrosstalkSources_855821_55.stderr parsl.dataflow.dflow DEBUG: Task 232 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 232 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 232 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 232 has memoization hash a18891cb32a2fee5dfa37b64c356095a parsl.dataflow.memoization INFO: Task 232 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca86ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 232 try 0 launched on executor multi with executor id 233 parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/a1000149-39e1-4570-ab13-bd66acde7170_isrForCrosstalkSources_855910_18.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/a1000149-39e1-4570-ab13-bd66acde7170_isrForCrosstalkSources_855910_18.stderr parsl.dataflow.dflow DEBUG: Task 233 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 233 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 233 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 233 has memoization hash baa0b2cc743ff8f551255e467a646433 parsl.dataflow.memoization INFO: Task 233 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca86de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 233 try 0 launched on executor multi with executor id 234 parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/780cf3cb-4093-4ebb-ae7a-dea4b7d97f25_isrForCrosstalkSources_855911_8.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/780cf3cb-4093-4ebb-ae7a-dea4b7d97f25_isrForCrosstalkSources_855911_8.stderr parsl.dataflow.dflow DEBUG: Task 234 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 234 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 234 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 234 has memoization hash 7a30b2184473938f85c7279d05d88a0d parsl.dataflow.memoization INFO: Task 234 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca86d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 234 try 0 launched on executor multi with executor id 235 parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/6eb70db8-fa64-42d3-88bb-32dea8d37670_isrForCrosstalkSources_855822_47.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/6eb70db8-fa64-42d3-88bb-32dea8d37670_isrForCrosstalkSources_855822_47.stderr parsl.dataflow.dflow DEBUG: Task 235 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 235 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 235 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 235 has memoization hash 46174b62094407a78f6de0392cdcc892 parsl.dataflow.memoization INFO: Task 235 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca86e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 235 try 0 launched on executor multi with executor id 236 parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/b2ca9ba2-a204-42d1-9a4f-70015053a679_isrForCrosstalkSources_855911_32.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/b2ca9ba2-a204-42d1-9a4f-70015053a679_isrForCrosstalkSources_855911_32.stderr parsl.dataflow.dflow DEBUG: Task 236 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 236 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 236 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 236 has memoization hash c48ed28e36aae8f9afa0ad87a137b644 parsl.dataflow.memoization INFO: Task 236 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca86f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 236 try 0 launched on executor multi with executor id 237 parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/55f226b8-143e-491f-9576-bcf776b95423_isrForCrosstalkSources_855909_48.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/55f226b8-143e-491f-9576-bcf776b95423_isrForCrosstalkSources_855909_48.stderr parsl.dataflow.dflow DEBUG: Task 237 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 237 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 237 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 237 has memoization hash d8a8d76bb53a6ff7cf0a53438ecdc426 parsl.dataflow.memoization INFO: Task 237 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca86fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 237 try 0 launched on executor multi with executor id 238 parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/19891b45-e8fd-4ce9-865c-470b7ca1c5e3_isrForCrosstalkSources_855911_56.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/19891b45-e8fd-4ce9-865c-470b7ca1c5e3_isrForCrosstalkSources_855911_56.stderr parsl.dataflow.dflow DEBUG: Task 238 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 238 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 238 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 238 has memoization hash b1457d72f465da99a7c5b409a834522a parsl.dataflow.memoization INFO: Task 238 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca87100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 238 try 0 launched on executor multi with executor id 239 parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/b0b34994-98d2-4065-8e9e-04e7501c51df_isrForCrosstalkSources_855822_16.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/b0b34994-98d2-4065-8e9e-04e7501c51df_isrForCrosstalkSources_855822_16.stderr parsl.dataflow.dflow DEBUG: Task 239 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 239 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 239 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 239 has memoization hash c7d855b0ff48e45c79e1ca3fd387d570 parsl.dataflow.memoization INFO: Task 239 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca87060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 239 try 0 launched on executor multi with executor id 240 parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/6a730bb7-2ff1-4f1a-9d90-7a369315a44b_isrForCrosstalkSources_855820_26.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/6a730bb7-2ff1-4f1a-9d90-7a369315a44b_isrForCrosstalkSources_855820_26.stderr parsl.dataflow.dflow DEBUG: Task 240 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 240 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 240 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 240 has memoization hash 6b61d8b7e1fcdec83a1c511450a85fd2 parsl.dataflow.memoization INFO: Task 240 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca871a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 240 try 0 launched on executor multi with executor id 241 parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/1a78c147-5622-4235-94e8-30d4cb307b8d_isrForCrosstalkSources_855820_35.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/1a78c147-5622-4235-94e8-30d4cb307b8d_isrForCrosstalkSources_855820_35.stderr parsl.dataflow.dflow DEBUG: Task 241 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 241 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 241 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 241 has memoization hash 18924f260b2a026a97db62dfa73681f7 parsl.dataflow.memoization INFO: Task 241 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca87240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 241 try 0 launched on executor multi with executor id 242 parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/6e67293a-0ac9-4640-b10c-b40216c2cb22_isrForCrosstalkSources_855821_19.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/6e67293a-0ac9-4640-b10c-b40216c2cb22_isrForCrosstalkSources_855821_19.stderr parsl.dataflow.dflow DEBUG: Task 242 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 242 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 242 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 242 has memoization hash 635927ca59f516e6620d3114c28920f8 parsl.dataflow.memoization INFO: Task 242 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca872e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 242 try 0 launched on executor multi with executor id 243 parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/671a4b04-d910-4966-b110-43a9709669fe_isrForCrosstalkSources_855821_41.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/671a4b04-d910-4966-b110-43a9709669fe_isrForCrosstalkSources_855821_41.stderr parsl.dataflow.dflow DEBUG: Task 243 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 243 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 243 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 243 has memoization hash 96cb37ff19c1e4fae677195df23dd260 parsl.dataflow.memoization INFO: Task 243 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca87420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 243 try 0 launched on executor multi with executor id 244 parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/af25f7b5-27f5-4d27-bdcb-2e3c3b4e63ec_isrForCrosstalkSources_855911_21.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/af25f7b5-27f5-4d27-bdcb-2e3c3b4e63ec_isrForCrosstalkSources_855911_21.stderr parsl.dataflow.dflow DEBUG: Task 244 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 244 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 244 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 244 has memoization hash a1ba7a5cae771909a9db534e8a9576aa parsl.dataflow.memoization INFO: Task 244 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca87380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 244 try 0 launched on executor multi with executor id 245 parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/c902e118-4d11-4fda-b8c0-71438b9f3651_isrForCrosstalkSources_855909_11.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/c902e118-4d11-4fda-b8c0-71438b9f3651_isrForCrosstalkSources_855909_11.stderr parsl.dataflow.dflow DEBUG: Task 245 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 245 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 245 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 245 has memoization hash d3ffc69088e04ec08b190a5f8d3d8b9f parsl.dataflow.memoization INFO: Task 245 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca874c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 245 try 0 launched on executor multi with executor id 246 parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/fedcac09-dbb8-4cbe-b21a-35673f0ac727_isrForCrosstalkSources_855820_11.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/fedcac09-dbb8-4cbe-b21a-35673f0ac727_isrForCrosstalkSources_855820_11.stderr parsl.dataflow.dflow DEBUG: Task 246 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 246 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 246 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 246 has memoization hash 633dcc9e25f85728bc6d71694172d24e parsl.dataflow.memoization INFO: Task 246 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca87560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 246 try 0 launched on executor multi with executor id 247 parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/d42b9f39-6104-4398-85f1-4de0b57ad068_isrForCrosstalkSources_855821_28.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/d42b9f39-6104-4398-85f1-4de0b57ad068_isrForCrosstalkSources_855821_28.stderr parsl.dataflow.dflow DEBUG: Task 247 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 247 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 247 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 247 has memoization hash 409be12ae56016383709e63626f14b3d parsl.dataflow.memoization INFO: Task 247 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca87600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 247 try 0 launched on executor multi with executor id 248 parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/bd6e23c6-f3f8-4b53-9c9e-4242db2a6a99_isrForCrosstalkSources_855910_57.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/bd6e23c6-f3f8-4b53-9c9e-4242db2a6a99_isrForCrosstalkSources_855910_57.stderr parsl.dataflow.dflow DEBUG: Task 248 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 248 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 248 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 248 has memoization hash 35ea7d9a4c5484abac34aeb9afeca9ce parsl.dataflow.memoization INFO: Task 248 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca87740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 248 try 0 launched on executor multi with executor id 249 parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/e12df680-210c-42d6-8e38-f2d5640e4683_isrForCrosstalkSources_855821_37.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/e12df680-210c-42d6-8e38-f2d5640e4683_isrForCrosstalkSources_855821_37.stderr parsl.dataflow.dflow DEBUG: Task 249 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 249 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 249 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 249 has memoization hash 0e76f3648ee03743a4b214b882e8387a parsl.dataflow.memoization INFO: Task 249 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca876a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 249 try 0 launched on executor multi with executor id 250 parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/740059b6-a066-484f-8ac3-f5fc2a765adf_isrForCrosstalkSources_855910_3.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/740059b6-a066-484f-8ac3-f5fc2a765adf_isrForCrosstalkSources_855910_3.stderr parsl.dataflow.dflow DEBUG: Task 250 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 250 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 250 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 250 has memoization hash e962419f14c59f0985308b072b045095 parsl.dataflow.memoization INFO: Task 250 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca877e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 250 try 0 launched on executor multi with executor id 251 parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/a52868bf-f587-46cc-88c7-4c0360e106ec_isrForCrosstalkSources_855911_22.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/a52868bf-f587-46cc-88c7-4c0360e106ec_isrForCrosstalkSources_855911_22.stderr parsl.dataflow.dflow DEBUG: Task 251 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 251 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 251 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 251 has memoization hash c22a2b40f728ab2d05f8b3a6f10da05c parsl.dataflow.memoization INFO: Task 251 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca87880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 251 try 0 launched on executor multi with executor id 252 parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/5a5bb5bd-e724-489a-b9db-3b0939427895_isrForCrosstalkSources_855910_27.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/5a5bb5bd-e724-489a-b9db-3b0939427895_isrForCrosstalkSources_855910_27.stderr parsl.dataflow.dflow DEBUG: Task 252 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 252 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 252 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 252 has memoization hash 7fdfc0085e25dfa1d701eea168d1981e parsl.dataflow.memoization INFO: Task 252 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca87920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 252 try 0 launched on executor multi with executor id 253 parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/f8ad3f7b-ce3d-4505-81fc-f0f76fc7a518_isrForCrosstalkSources_855820_30.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/f8ad3f7b-ce3d-4505-81fc-f0f76fc7a518_isrForCrosstalkSources_855820_30.stderr parsl.dataflow.dflow DEBUG: Task 253 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 253 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 253 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 253 has memoization hash 64b1cab9d2c693e9a4cd002fcbd3a3a1 parsl.dataflow.memoization INFO: Task 253 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca87a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 253 try 0 launched on executor multi with executor id 254 parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/51d755f8-588c-4187-9ad0-7be446d4c936_isrForCrosstalkSources_855820_8.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/51d755f8-588c-4187-9ad0-7be446d4c936_isrForCrosstalkSources_855820_8.stderr parsl.dataflow.dflow DEBUG: Task 254 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 254 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 254 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 254 has memoization hash f9675f37b63d33f6f23d44e30772124f parsl.dataflow.memoization INFO: Task 254 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca879c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 254 try 0 launched on executor multi with executor id 255 parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/c825b944-98ef-46c2-999f-b7a9bae78744_isrForCrosstalkSources_855820_44.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/c825b944-98ef-46c2-999f-b7a9bae78744_isrForCrosstalkSources_855820_44.stderr parsl.dataflow.dflow DEBUG: Task 255 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 255 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 255 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 255 has memoization hash 6bd04404ea4f85e9722a879c8e61a5e5 parsl.dataflow.memoization INFO: Task 255 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca87b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 255 try 0 launched on executor multi with executor id 256 parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/db56d21d-3cd4-4a0f-a281-252d0afbdcde_isrForCrosstalkSources_855909_14.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/db56d21d-3cd4-4a0f-a281-252d0afbdcde_isrForCrosstalkSources_855909_14.stderr parsl.dataflow.dflow DEBUG: Task 256 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 256 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 256 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 256 has memoization hash 2b6c2fc05eb470378ed534a501a8a55c parsl.dataflow.memoization INFO: Task 256 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca87ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 256 try 0 launched on executor multi with executor id 257 parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/c7ca1547-1a7e-45aa-a520-8138bc98c811_isrForCrosstalkSources_855821_61.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/c7ca1547-1a7e-45aa-a520-8138bc98c811_isrForCrosstalkSources_855821_61.stderr parsl.dataflow.dflow DEBUG: Task 257 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 257 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 257 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 257 has memoization hash 2fc38dcee97b961a95758c64b80dd6d6 parsl.dataflow.memoization INFO: Task 257 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca87c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 257 try 0 launched on executor multi with executor id 258 parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/eb391d8d-d471-4a43-88a1-51502968a348_isrForCrosstalkSources_855822_1.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/eb391d8d-d471-4a43-88a1-51502968a348_isrForCrosstalkSources_855822_1.stderr parsl.dataflow.dflow DEBUG: Task 258 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 258 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 258 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 258 has memoization hash f5662247dfce05540a7cd5c9916f3345 parsl.dataflow.memoization INFO: Task 258 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca87d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 258 try 0 launched on executor multi with executor id 259 parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/f5ca4899-caf2-4a24-978b-90aef5dc243e_isrForCrosstalkSources_855821_29.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/f5ca4899-caf2-4a24-978b-90aef5dc243e_isrForCrosstalkSources_855821_29.stderr parsl.dataflow.dflow DEBUG: Task 259 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 259 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 259 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 259 has memoization hash 832fe61760a3d275415b16271f58e6e7 parsl.dataflow.memoization INFO: Task 259 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca87ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 259 try 0 launched on executor multi with executor id 260 parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/79e36655-1696-40ca-8161-193745171f30_isrForCrosstalkSources_855821_57.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/79e36655-1696-40ca-8161-193745171f30_isrForCrosstalkSources_855821_57.stderr parsl.dataflow.dflow DEBUG: Task 260 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 260 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 260 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 260 has memoization hash d39753135d8bc3aec65b89dca7a1be65 parsl.dataflow.memoization INFO: Task 260 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca87e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 260 try 0 launched on executor multi with executor id 261 parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/da5faeda-d22f-4b6c-83fa-413a5272ea12_isrForCrosstalkSources_855822_11.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/da5faeda-d22f-4b6c-83fa-413a5272ea12_isrForCrosstalkSources_855822_11.stderr parsl.dataflow.dflow DEBUG: Task 261 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 261 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 261 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 261 has memoization hash 5d6778eed17f290cfb093d98f617f219 parsl.dataflow.memoization INFO: Task 261 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca87ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 261 try 0 launched on executor multi with executor id 262 parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/9700f98f-94e9-4036-83db-69de371cebd4_isrForCrosstalkSources_855820_19.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/9700f98f-94e9-4036-83db-69de371cebd4_isrForCrosstalkSources_855820_19.stderr parsl.dataflow.dflow DEBUG: Task 262 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 262 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 262 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 262 has memoization hash ecf54c57e80c4f47f27905e5c0b7a5fb parsl.dataflow.memoization INFO: Task 262 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca87f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 262 try 0 launched on executor multi with executor id 263 parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/52fcfbf8-9559-4cf4-b6d2-b561115bedf8_isrForCrosstalkSources_855822_43.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/52fcfbf8-9559-4cf4-b6d2-b561115bedf8_isrForCrosstalkSources_855822_43.stderr parsl.dataflow.dflow DEBUG: Task 263 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 263 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 263 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 263 has memoization hash 282021e2dd2b25f708dc9950f1761c18 parsl.dataflow.memoization INFO: Task 263 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dca84040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 263 try 0 launched on executor multi with executor id 264 parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/1486ec87-2b6f-4cfe-b27b-ebafb46995bb_isrForCrosstalkSources_855909_3.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/1486ec87-2b6f-4cfe-b27b-ebafb46995bb_isrForCrosstalkSources_855909_3.stderr parsl.dataflow.dflow DEBUG: Task 264 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 264 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 264 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 264 has memoization hash 55eb33ffa641d7f8ea4e7ea972d4b72f parsl.dataflow.memoization INFO: Task 264 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb10040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 264 try 0 launched on executor multi with executor id 265 parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/6cb51672-26d7-452f-bb94-2880e713542a_isrForCrosstalkSources_855911_54.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/6cb51672-26d7-452f-bb94-2880e713542a_isrForCrosstalkSources_855911_54.stderr parsl.dataflow.dflow DEBUG: Task 265 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 265 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 265 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 265 has memoization hash a337ffb91de0888a20ec67858b751b96 parsl.dataflow.memoization INFO: Task 265 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb10180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 265 try 0 launched on executor multi with executor id 266 parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/c8629cb5-601d-44ed-987c-4c42d22fee26_isrForCrosstalkSources_855909_31.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/c8629cb5-601d-44ed-987c-4c42d22fee26_isrForCrosstalkSources_855909_31.stderr parsl.dataflow.dflow DEBUG: Task 266 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 266 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 266 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 266 has memoization hash 2d6dd667ca13c2e78ed6f9e094c10d28 parsl.dataflow.memoization INFO: Task 266 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb10220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 266 try 0 launched on executor multi with executor id 267 parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/8ec91fd1-fdc6-4c00-be1a-20bde715d59c_isrForCrosstalkSources_855821_36.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/8ec91fd1-fdc6-4c00-be1a-20bde715d59c_isrForCrosstalkSources_855821_36.stderr parsl.dataflow.dflow DEBUG: Task 267 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 267 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 267 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 267 has memoization hash 8541532df6063a31a0045e241a6fa24e parsl.dataflow.memoization INFO: Task 267 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb102c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 267 try 0 launched on executor multi with executor id 268 parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/57a0d132-9bf2-429a-8ce1-f43040a878e5_isrForCrosstalkSources_855909_55.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/57a0d132-9bf2-429a-8ce1-f43040a878e5_isrForCrosstalkSources_855909_55.stderr parsl.dataflow.dflow DEBUG: Task 268 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 268 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 268 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 268 has memoization hash acd8ea4cfd0f2bb4aeedc02dd4a04b4a parsl.dataflow.memoization INFO: Task 268 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb10400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 268 try 0 launched on executor multi with executor id 269 parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/cddd1985-a124-4ef2-a88f-a93a93da4608_isrForCrosstalkSources_855910_56.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/cddd1985-a124-4ef2-a88f-a93a93da4608_isrForCrosstalkSources_855910_56.stderr parsl.dataflow.dflow DEBUG: Task 269 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 269 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 269 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 269 has memoization hash 063759d5639aa7f57ec0d196ede0d147 parsl.dataflow.memoization INFO: Task 269 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb10360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 269 try 0 launched on executor multi with executor id 270 parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/f24f9c17-b202-4416-a26a-f27644492ce7_isrForCrosstalkSources_855910_26.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/f24f9c17-b202-4416-a26a-f27644492ce7_isrForCrosstalkSources_855910_26.stderr parsl.dataflow.dflow DEBUG: Task 270 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 270 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 270 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 270 has memoization hash 338e2bcc22edeb5d43bcc2765dc5293c parsl.dataflow.memoization INFO: Task 270 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb104a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 270 try 0 launched on executor multi with executor id 271 parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/eddbd2d3-4c81-482e-b84f-1a4ba689bc36_isrForCrosstalkSources_855911_43.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/eddbd2d3-4c81-482e-b84f-1a4ba689bc36_isrForCrosstalkSources_855911_43.stderr parsl.dataflow.dflow DEBUG: Task 271 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 271 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 271 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 271 has memoization hash f940ffe5d331c77441abc9ae954db895 parsl.dataflow.memoization INFO: Task 271 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb10540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 271 try 0 launched on executor multi with executor id 272 parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/e6a2934d-a52e-4d8b-b913-3b6dee1ad150_isrForCrosstalkSources_855911_44.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/e6a2934d-a52e-4d8b-b913-3b6dee1ad150_isrForCrosstalkSources_855911_44.stderr parsl.dataflow.dflow DEBUG: Task 272 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 272 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 272 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 272 has memoization hash cc227710b4c67670939f3cb3fef8834c parsl.dataflow.memoization INFO: Task 272 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb105e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 272 try 0 launched on executor multi with executor id 273 parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/103327fd-551d-403c-b6a1-d51897922402_isrForCrosstalkSources_855822_57.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/103327fd-551d-403c-b6a1-d51897922402_isrForCrosstalkSources_855822_57.stderr parsl.dataflow.dflow DEBUG: Task 273 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 273 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 273 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 273 has memoization hash 0a992c16a3800ca83b3b2f0e9b7bc1f1 parsl.dataflow.memoization INFO: Task 273 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb10720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 273 try 0 launched on executor multi with executor id 274 parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/5a6ec4e4-3b55-47a6-948b-00b62bbc90a9_isrForCrosstalkSources_855909_58.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/5a6ec4e4-3b55-47a6-948b-00b62bbc90a9_isrForCrosstalkSources_855909_58.stderr parsl.dataflow.dflow DEBUG: Task 274 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 274 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 274 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 274 has memoization hash 9b555937498fb4c6d1e094ae245ec290 parsl.dataflow.memoization INFO: Task 274 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb10680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 274 try 0 launched on executor multi with executor id 275 parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/3e0bfff9-0a83-436d-8efb-962177ee6a7c_isrForCrosstalkSources_855820_58.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/3e0bfff9-0a83-436d-8efb-962177ee6a7c_isrForCrosstalkSources_855820_58.stderr parsl.dataflow.dflow DEBUG: Task 275 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 275 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 275 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 275 has memoization hash 730c99d63433688a19005a99874b925a parsl.dataflow.memoization INFO: Task 275 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb107c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 275 try 0 launched on executor multi with executor id 276 parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/2aa18c98-1195-4d3c-bcf2-cd188c95f409_isrForCrosstalkSources_855910_30.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/2aa18c98-1195-4d3c-bcf2-cd188c95f409_isrForCrosstalkSources_855910_30.stderr parsl.dataflow.dflow DEBUG: Task 276 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 276 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 276 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 276 has memoization hash 5d588906a261f9a1ae0f95ecdbfa3154 parsl.dataflow.memoization INFO: Task 276 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb10860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 276 try 0 launched on executor multi with executor id 277 parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/b1faafd2-5206-4c80-a812-245cdf1061ce_isrForCrosstalkSources_855911_3.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/b1faafd2-5206-4c80-a812-245cdf1061ce_isrForCrosstalkSources_855911_3.stderr parsl.dataflow.dflow DEBUG: Task 277 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 277 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 277 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 277 has memoization hash d592f4ad700606f485abb90d4c95be08 parsl.dataflow.memoization INFO: Task 277 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb10900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 277 try 0 launched on executor multi with executor id 278 parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/ef3fb890-b96a-4580-92b9-3b0cd74379c2_isrForCrosstalkSources_855909_45.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/ef3fb890-b96a-4580-92b9-3b0cd74379c2_isrForCrosstalkSources_855909_45.stderr parsl.dataflow.dflow DEBUG: Task 278 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 278 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 278 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 278 has memoization hash 7cf84fedbc0561a9bcfb5b266230ef51 parsl.dataflow.memoization INFO: Task 278 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb10a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 278 try 0 launched on executor multi with executor id 279 parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/f773b98e-dd0c-4976-a583-f83bdf4bb20f_isrForCrosstalkSources_855910_15.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/f773b98e-dd0c-4976-a583-f83bdf4bb20f_isrForCrosstalkSources_855910_15.stderr parsl.dataflow.dflow DEBUG: Task 279 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 279 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 279 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 279 has memoization hash e5d29a4f360d2f9075424bea744914e7 parsl.dataflow.memoization INFO: Task 279 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb109a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 279 try 0 launched on executor multi with executor id 280 parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/1201a4e6-5ab4-4211-880c-5db79910c576_isrForCrosstalkSources_855911_23.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/1201a4e6-5ab4-4211-880c-5db79910c576_isrForCrosstalkSources_855911_23.stderr parsl.dataflow.dflow DEBUG: Task 280 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 280 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 280 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 280 has memoization hash 6a181b23a81211cbabe4368f39c67e94 parsl.dataflow.memoization INFO: Task 280 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb10ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 280 try 0 launched on executor multi with executor id 281 parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/4ab750c2-2134-4425-b3c2-f12bf2420cca_isrForCrosstalkSources_855820_15.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/4ab750c2-2134-4425-b3c2-f12bf2420cca_isrForCrosstalkSources_855820_15.stderr parsl.dataflow.dflow DEBUG: Task 281 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 281 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 281 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 281 has memoization hash f3b460ffc085e90a501c5af03b5fc417 parsl.dataflow.memoization INFO: Task 281 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb10b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 281 try 0 launched on executor multi with executor id 282 parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/7e540e51-292b-4c64-a422-86035fa212cd_isrForCrosstalkSources_855910_1.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/7e540e51-292b-4c64-a422-86035fa212cd_isrForCrosstalkSources_855910_1.stderr parsl.dataflow.dflow DEBUG: Task 282 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 282 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 282 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 282 has memoization hash 4851630223bce9cb78b4f540a3015c03 parsl.dataflow.memoization INFO: Task 282 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb10c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 282 try 0 launched on executor multi with executor id 283 parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/d4087a6b-d3dd-4d72-8032-0b228dfac24a_isrForCrosstalkSources_855821_7.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/d4087a6b-d3dd-4d72-8032-0b228dfac24a_isrForCrosstalkSources_855821_7.stderr parsl.dataflow.dflow DEBUG: Task 283 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 283 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 283 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 283 has memoization hash 3b3c098d7fed52cc3555a8778d54be6a parsl.dataflow.memoization INFO: Task 283 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb10d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 283 try 0 launched on executor multi with executor id 284 parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/3a1a6172-5a90-4b08-9fff-e859baa02dc2_isrForCrosstalkSources_855911_17.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/3a1a6172-5a90-4b08-9fff-e859baa02dc2_isrForCrosstalkSources_855911_17.stderr parsl.dataflow.dflow DEBUG: Task 284 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 284 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 284 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 284 has memoization hash ada684595c53eb9495844296cbd04bf0 parsl.dataflow.memoization INFO: Task 284 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb10cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 284 try 0 launched on executor multi with executor id 285 parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/ee703511-563b-447a-a833-a761c2037225_isrForCrosstalkSources_855909_62.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/ee703511-563b-447a-a833-a761c2037225_isrForCrosstalkSources_855909_62.stderr parsl.dataflow.dflow DEBUG: Task 285 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 285 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 285 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 285 has memoization hash c69664592d63539e1082df569ec66247 parsl.dataflow.memoization INFO: Task 285 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb10e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 285 try 0 launched on executor multi with executor id 286 parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/fd59d7e6-6502-431c-b44c-fb4fc6f07d15_isrForCrosstalkSources_855822_44.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/fd59d7e6-6502-431c-b44c-fb4fc6f07d15_isrForCrosstalkSources_855822_44.stderr parsl.dataflow.dflow DEBUG: Task 286 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 286 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 286 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 286 has memoization hash fef66d47c178ae0f6f0629b6fa83e1d2 parsl.dataflow.memoization INFO: Task 286 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb10ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 286 try 0 launched on executor multi with executor id 287 parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/c4251f23-8b4b-414c-84a6-e624c42d63ad_isrForCrosstalkSources_855911_49.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/c4251f23-8b4b-414c-84a6-e624c42d63ad_isrForCrosstalkSources_855911_49.stderr parsl.dataflow.dflow DEBUG: Task 287 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 287 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 287 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 287 has memoization hash 44d44b06c4faf0e6b2ec9b8326cc0b4f parsl.dataflow.memoization INFO: Task 287 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb10f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 287 try 0 launched on executor multi with executor id 288 parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/911eb74c-ffee-42e4-bbcc-7c05fcaecefe_isrForCrosstalkSources_855909_50.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/911eb74c-ffee-42e4-bbcc-7c05fcaecefe_isrForCrosstalkSources_855909_50.stderr parsl.dataflow.dflow DEBUG: Task 288 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 288 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 288 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 288 has memoization hash 5ca2582ebee230e86d32a07b64841d6a parsl.dataflow.memoization INFO: Task 288 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb11080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 288 try 0 launched on executor multi with executor id 289 parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/5bb84ce5-cd2a-4c62-b98d-bd775d90bb24_isrForCrosstalkSources_855820_9.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/5bb84ce5-cd2a-4c62-b98d-bd775d90bb24_isrForCrosstalkSources_855820_9.stderr parsl.dataflow.dflow DEBUG: Task 289 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 289 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 289 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 289 has memoization hash abd1fbfc28232298711a382ecf23fa13 parsl.dataflow.memoization INFO: Task 289 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb10fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 289 try 0 launched on executor multi with executor id 290 parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/43c3d0ee-a909-4279-b0d3-4c67c0955a76_isrForCrosstalkSources_855820_18.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/43c3d0ee-a909-4279-b0d3-4c67c0955a76_isrForCrosstalkSources_855820_18.stderr parsl.dataflow.dflow DEBUG: Task 290 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 290 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 290 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 290 has memoization hash af9a515f085dc64cc5f5649041033024 parsl.dataflow.memoization INFO: Task 290 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb11120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 290 try 0 launched on executor multi with executor id 291 parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/1e3e312a-9bc7-49e1-9c1e-671fa2bc0746_isrForCrosstalkSources_855821_22.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/1e3e312a-9bc7-49e1-9c1e-671fa2bc0746_isrForCrosstalkSources_855821_22.stderr parsl.dataflow.dflow DEBUG: Task 291 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 291 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 291 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 291 has memoization hash e14b889ef22080f73a75bd6ae594ee5c parsl.dataflow.memoization INFO: Task 291 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb111c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 291 try 0 launched on executor multi with executor id 292 parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/cc77215b-6118-4b55-b537-ba7abfa1ab95_isrForCrosstalkSources_855820_33.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/cc77215b-6118-4b55-b537-ba7abfa1ab95_isrForCrosstalkSources_855820_33.stderr parsl.dataflow.dflow DEBUG: Task 292 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 292 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 292 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 292 has memoization hash d48396159640a7cb5a7f6e2da8071e72 parsl.dataflow.memoization INFO: Task 292 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb11260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 292 try 0 launched on executor multi with executor id 293 parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/3acd1299-7784-4362-86dc-78bc156b3354_isrForCrosstalkSources_855822_46.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/3acd1299-7784-4362-86dc-78bc156b3354_isrForCrosstalkSources_855822_46.stderr parsl.dataflow.dflow DEBUG: Task 293 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 293 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 293 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 293 has memoization hash 71f815faa129bb5a4d5fc4d3c5b7fb1f parsl.dataflow.memoization INFO: Task 293 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb113a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 293 try 0 launched on executor multi with executor id 294 parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/4af67321-bdc6-4bfd-a535-10c54dda467d_isrForCrosstalkSources_855909_27.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/4af67321-bdc6-4bfd-a535-10c54dda467d_isrForCrosstalkSources_855909_27.stderr parsl.dataflow.dflow DEBUG: Task 294 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 294 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 294 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 294 has memoization hash ee18e9dcd675362d76e039f2c0a4b5ed parsl.dataflow.memoization INFO: Task 294 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb11300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 294 try 0 launched on executor multi with executor id 295 parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/9c4ed7c2-2683-4dc7-b1f6-3cdf22fdf81a_isrForCrosstalkSources_855911_51.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/9c4ed7c2-2683-4dc7-b1f6-3cdf22fdf81a_isrForCrosstalkSources_855911_51.stderr parsl.dataflow.dflow DEBUG: Task 295 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 295 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 295 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 295 has memoization hash 223f176455aa6e2ac52cbbe17f0e87e8 parsl.dataflow.memoization INFO: Task 295 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb11440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 295 try 0 launched on executor multi with executor id 296 parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/9c26958f-bcd6-49c4-bd32-0c7eeeaa1836_isrForCrosstalkSources_855822_23.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/9c26958f-bcd6-49c4-bd32-0c7eeeaa1836_isrForCrosstalkSources_855822_23.stderr parsl.dataflow.dflow DEBUG: Task 296 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 296 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 296 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 296 has memoization hash 37fc4b160f2c3747ba7dc55caf34ec42 parsl.dataflow.memoization INFO: Task 296 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb114e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 296 try 0 launched on executor multi with executor id 297 parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/070a9efb-59d6-4a61-829e-8b0f80542380_isrForCrosstalkSources_855820_47.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/070a9efb-59d6-4a61-829e-8b0f80542380_isrForCrosstalkSources_855820_47.stderr parsl.dataflow.dflow DEBUG: Task 297 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 297 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 297 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 297 has memoization hash b2157342f1ec0a1f78db96715a680b83 parsl.dataflow.memoization INFO: Task 297 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb11580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 297 try 0 launched on executor multi with executor id 298 parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/846a9bb2-2d8e-4a2e-8fe8-5c94cdd884f9_isrForCrosstalkSources_855822_53.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/846a9bb2-2d8e-4a2e-8fe8-5c94cdd884f9_isrForCrosstalkSources_855822_53.stderr parsl.dataflow.dflow DEBUG: Task 298 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 298 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 298 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 298 has memoization hash be2315c283aa0d01faa96eaf9acaad73 parsl.dataflow.memoization INFO: Task 298 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb116c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 298 try 0 launched on executor multi with executor id 299 parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/947364fa-423b-4b1e-8aed-726cb44bad7c_isrForCrosstalkSources_855822_41.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/947364fa-423b-4b1e-8aed-726cb44bad7c_isrForCrosstalkSources_855822_41.stderr parsl.dataflow.dflow DEBUG: Task 299 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 299 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 299 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 299 has memoization hash 7b7760d4b247a48fbf5a42bc8bf4cf00 parsl.dataflow.memoization INFO: Task 299 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb11620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 299 try 0 launched on executor multi with executor id 300 parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/6f27cc4c-ff62-4c4d-b5f8-e11d6ad73c0e_isrForCrosstalkSources_855822_60.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/6f27cc4c-ff62-4c4d-b5f8-e11d6ad73c0e_isrForCrosstalkSources_855822_60.stderr parsl.dataflow.dflow DEBUG: Task 300 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 300 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 300 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 300 has memoization hash 8aa82888f6ea6b537c546b58c79e1be5 parsl.dataflow.memoization INFO: Task 300 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb11760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 300 try 0 launched on executor multi with executor id 301 parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/e169bfe2-38b0-4407-9fdb-6e81d5747e23_isrForCrosstalkSources_855822_3.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/e169bfe2-38b0-4407-9fdb-6e81d5747e23_isrForCrosstalkSources_855822_3.stderr parsl.dataflow.dflow DEBUG: Task 301 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 301 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 301 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 301 has memoization hash 10856ee910e83712753c46e59ad8d650 parsl.dataflow.memoization INFO: Task 301 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb11800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 301 try 0 launched on executor multi with executor id 302 parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/b88e8079-1f50-4e87-b00f-2e6580663231_isrForCrosstalkSources_855911_13.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/b88e8079-1f50-4e87-b00f-2e6580663231_isrForCrosstalkSources_855911_13.stderr parsl.dataflow.dflow DEBUG: Task 302 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 302 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 302 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 302 has memoization hash 13e1862f2fa12b4f759ca3efbe9441f5 parsl.dataflow.memoization INFO: Task 302 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb118a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 302 try 0 launched on executor multi with executor id 303 parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/4ed14335-6146-4037-9372-29ac05d84cd1_isrForCrosstalkSources_855909_42.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/4ed14335-6146-4037-9372-29ac05d84cd1_isrForCrosstalkSources_855909_42.stderr parsl.dataflow.dflow DEBUG: Task 303 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 303 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 303 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 303 has memoization hash 640440fe7a408c6975b660f3ee0cb73f parsl.dataflow.memoization INFO: Task 303 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb119e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 303 try 0 launched on executor multi with executor id 304 parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/272e941f-5e06-451b-895f-9e10610d65bc_isrForCrosstalkSources_855822_52.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/272e941f-5e06-451b-895f-9e10610d65bc_isrForCrosstalkSources_855822_52.stderr parsl.dataflow.dflow DEBUG: Task 304 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 304 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 304 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 304 has memoization hash 7b85d154e314966bde6028d21886f8b0 parsl.dataflow.memoization INFO: Task 304 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb11940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 304 try 0 launched on executor multi with executor id 305 parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/c5122f89-4e7a-48b3-bfeb-3330c79668d4_isrForCrosstalkSources_855910_31.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/c5122f89-4e7a-48b3-bfeb-3330c79668d4_isrForCrosstalkSources_855910_31.stderr parsl.dataflow.dflow DEBUG: Task 305 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 305 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 305 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 305 has memoization hash 33cf69813c75aa9d075a6d94c476693e parsl.dataflow.memoization INFO: Task 305 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb11a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 305 try 0 launched on executor multi with executor id 306 parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/671779e8-d5a3-4e86-9d24-046deac10258_isrForCrosstalkSources_855910_41.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/671779e8-d5a3-4e86-9d24-046deac10258_isrForCrosstalkSources_855910_41.stderr parsl.dataflow.dflow DEBUG: Task 306 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 306 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 306 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 306 has memoization hash 9ebc89f41ed1ce7350736cbc9df2be7a parsl.dataflow.memoization INFO: Task 306 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb11b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 306 try 0 launched on executor multi with executor id 307 parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/4923767e-2665-4ab4-b632-366b8412d0f0_isrForCrosstalkSources_855909_37.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/4923767e-2665-4ab4-b632-366b8412d0f0_isrForCrosstalkSources_855909_37.stderr parsl.dataflow.dflow DEBUG: Task 307 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 307 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 307 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 307 has memoization hash 7c98d4315c9ba38a3dd5086375cddf23 parsl.dataflow.memoization INFO: Task 307 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb11bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 307 try 0 launched on executor multi with executor id 308 parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/7f1b2711-b446-42d6-9baf-dc025e0c2fdc_isrForCrosstalkSources_855821_20.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/7f1b2711-b446-42d6-9baf-dc025e0c2fdc_isrForCrosstalkSources_855821_20.stderr parsl.dataflow.dflow DEBUG: Task 308 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 308 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 308 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 308 has memoization hash 98c91f1a78b282ec993a7b759f1cfc11 parsl.dataflow.memoization INFO: Task 308 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb11d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 308 try 0 launched on executor multi with executor id 309 parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/0fb4d167-ef8a-45a3-807f-9b6511112193_isrForCrosstalkSources_855821_46.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/0fb4d167-ef8a-45a3-807f-9b6511112193_isrForCrosstalkSources_855821_46.stderr parsl.dataflow.dflow DEBUG: Task 309 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 309 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 309 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 309 has memoization hash 628a0d291609a70b014bfe574b1a40fa parsl.dataflow.memoization INFO: Task 309 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb11c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 309 try 0 launched on executor multi with executor id 310 parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/dbac9698-3296-4eba-b99b-8b621dc720a8_isrForCrosstalkSources_855822_33.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/dbac9698-3296-4eba-b99b-8b621dc720a8_isrForCrosstalkSources_855822_33.stderr parsl.dataflow.dflow DEBUG: Task 310 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 310 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 310 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 310 has memoization hash e90af6750fee9e561dd5eead46029b8a parsl.dataflow.memoization INFO: Task 310 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb11da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 310 try 0 launched on executor multi with executor id 311 parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/48258bbe-16cf-4006-9bc4-757f7c7f766f_isrForCrosstalkSources_855820_40.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/48258bbe-16cf-4006-9bc4-757f7c7f766f_isrForCrosstalkSources_855820_40.stderr parsl.dataflow.dflow DEBUG: Task 311 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 311 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 311 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 311 has memoization hash 1f73fe7263d73348044b0d00419a26af parsl.dataflow.memoization INFO: Task 311 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb11e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 311 try 0 launched on executor multi with executor id 312 parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/51bba081-88b9-45ce-bb37-eced9738702e_isrForCrosstalkSources_855820_60.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/51bba081-88b9-45ce-bb37-eced9738702e_isrForCrosstalkSources_855820_60.stderr parsl.dataflow.dflow DEBUG: Task 312 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 312 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 312 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 312 has memoization hash 7dac7bbda376d47720085b523ab551e8 parsl.dataflow.memoization INFO: Task 312 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb11ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 312 try 0 launched on executor multi with executor id 313 parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/1b93d2e9-64a5-42d1-9c6c-ec0fd2d90029_isrForCrosstalkSources_855822_15.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/1b93d2e9-64a5-42d1-9c6c-ec0fd2d90029_isrForCrosstalkSources_855822_15.stderr parsl.dataflow.dflow DEBUG: Task 313 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 313 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 313 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 313 has memoization hash 629b0a5a34c78c1b497515c4246cc488 parsl.dataflow.memoization INFO: Task 313 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb12020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 313 try 0 launched on executor multi with executor id 314 parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/2e9cd747-e5fe-41d2-8aba-c31f5e728022_isrForCrosstalkSources_855911_20.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/2e9cd747-e5fe-41d2-8aba-c31f5e728022_isrForCrosstalkSources_855911_20.stderr parsl.dataflow.dflow DEBUG: Task 314 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 314 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 314 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 314 has memoization hash 80c000b9cc7af4bd542a2864e455e539 parsl.dataflow.memoization INFO: Task 314 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb11f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 314 try 0 launched on executor multi with executor id 315 parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/a76ab128-fc61-4f01-b5fb-18b2d72d3e9f_isrForCrosstalkSources_855911_41.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/a76ab128-fc61-4f01-b5fb-18b2d72d3e9f_isrForCrosstalkSources_855911_41.stderr parsl.dataflow.dflow DEBUG: Task 315 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 315 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 315 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 315 has memoization hash ce2cfc2a1eee061a35d1a556adee5bbd parsl.dataflow.memoization INFO: Task 315 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb120c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 315 try 0 launched on executor multi with executor id 316 parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/89a22a36-37a0-41f5-9d74-4b317896e716_isrForCrosstalkSources_855909_10.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/89a22a36-37a0-41f5-9d74-4b317896e716_isrForCrosstalkSources_855909_10.stderr parsl.dataflow.dflow DEBUG: Task 316 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 316 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 316 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 316 has memoization hash 7e23771124066b415631e3bd860db742 parsl.dataflow.memoization INFO: Task 316 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb12160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 316 try 0 launched on executor multi with executor id 317 parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/b70ecc56-2c66-4a77-854c-2a633f593b06_isrForCrosstalkSources_855910_61.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/b70ecc56-2c66-4a77-854c-2a633f593b06_isrForCrosstalkSources_855910_61.stderr parsl.dataflow.dflow DEBUG: Task 317 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 317 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 317 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 317 has memoization hash c953aa8b25a6a18fc5a8dfb08bce7cda parsl.dataflow.memoization INFO: Task 317 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb12200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 317 try 0 launched on executor multi with executor id 318 parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/9f562a1c-a34d-4761-822e-951dafa1e3c9_isrForCrosstalkSources_855822_58.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/9f562a1c-a34d-4761-822e-951dafa1e3c9_isrForCrosstalkSources_855822_58.stderr parsl.dataflow.dflow DEBUG: Task 318 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 318 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 318 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 318 has memoization hash e38d51246548a31fd23f4072c1315f88 parsl.dataflow.memoization INFO: Task 318 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb12340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 318 try 0 launched on executor multi with executor id 319 parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/f1be4191-34fe-44f6-8caf-43bea778bf0e_isrForCrosstalkSources_855820_42.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/f1be4191-34fe-44f6-8caf-43bea778bf0e_isrForCrosstalkSources_855820_42.stderr parsl.dataflow.dflow DEBUG: Task 319 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 319 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 319 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 319 has memoization hash d1701dd52fc1659046d7f77bf9dad962 parsl.dataflow.memoization INFO: Task 319 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb122a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 319 try 0 launched on executor multi with executor id 320 parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/21411830-0cbe-47ec-8eb8-1fe76d8d979b_isrForCrosstalkSources_855822_28.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/21411830-0cbe-47ec-8eb8-1fe76d8d979b_isrForCrosstalkSources_855822_28.stderr parsl.dataflow.dflow DEBUG: Task 320 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 320 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 320 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 320 has memoization hash 23ba82224fcfaa4ef7d5febffe03b3a5 parsl.dataflow.memoization INFO: Task 320 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb123e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 320 try 0 launched on executor multi with executor id 321 parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/b2d24210-8aff-4534-ae26-32851c9d0d3a_isrForCrosstalkSources_855822_51.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/b2d24210-8aff-4534-ae26-32851c9d0d3a_isrForCrosstalkSources_855822_51.stderr parsl.dataflow.dflow DEBUG: Task 321 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 321 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 321 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 321 has memoization hash 2502d337e45c4bf74fedd012d639ba6e parsl.dataflow.memoization INFO: Task 321 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb12480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 321 try 0 launched on executor multi with executor id 322 parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/68164347-c2db-4b41-ae26-1f550dea04e6_isrForCrosstalkSources_855909_29.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/68164347-c2db-4b41-ae26-1f550dea04e6_isrForCrosstalkSources_855909_29.stderr parsl.dataflow.dflow DEBUG: Task 322 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 322 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 322 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 322 has memoization hash 83bb969def82aff3389f5f4389ea40af parsl.dataflow.memoization INFO: Task 322 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb12520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 322 try 0 launched on executor multi with executor id 323 parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/e26889ea-3fcd-4f14-b90d-3f438be37630_isrForCrosstalkSources_855822_31.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/e26889ea-3fcd-4f14-b90d-3f438be37630_isrForCrosstalkSources_855822_31.stderr parsl.dataflow.dflow DEBUG: Task 323 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 323 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 323 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 323 has memoization hash 0568499a4332d2367005e1b3b5ebad60 parsl.dataflow.memoization INFO: Task 323 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb12660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 323 try 0 launched on executor multi with executor id 324 parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/48d0536a-6458-42b3-b4bb-5af632145962_isrForCrosstalkSources_855821_50.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/48d0536a-6458-42b3-b4bb-5af632145962_isrForCrosstalkSources_855821_50.stderr parsl.dataflow.dflow DEBUG: Task 324 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 324 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 324 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 324 has memoization hash 697cb0588943e4da5e50478c2fabf012 parsl.dataflow.memoization INFO: Task 324 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb125c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 324 try 0 launched on executor multi with executor id 325 parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/55514120-103d-49d2-bcbc-bfcb5696b06a_isrForCrosstalkSources_855820_16.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/55514120-103d-49d2-bcbc-bfcb5696b06a_isrForCrosstalkSources_855820_16.stderr parsl.dataflow.dflow DEBUG: Task 325 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 325 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 325 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 325 has memoization hash d31482d10a58b8ef59d749e16bfb31d4 parsl.dataflow.memoization INFO: Task 325 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb12700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 325 try 0 launched on executor multi with executor id 326 parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/7deb925d-4e10-4e82-be32-82f55af462e1_isrForCrosstalkSources_855820_48.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/7deb925d-4e10-4e82-be32-82f55af462e1_isrForCrosstalkSources_855820_48.stderr parsl.dataflow.dflow DEBUG: Task 326 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 326 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 326 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 326 has memoization hash 3537ceee33aa614472d7376940f431e8 parsl.dataflow.memoization INFO: Task 326 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb127a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 326 try 0 launched on executor multi with executor id 327 parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/08556b91-46f2-4336-9838-1cf9dc515561_isrForCrosstalkSources_855822_35.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/08556b91-46f2-4336-9838-1cf9dc515561_isrForCrosstalkSources_855822_35.stderr parsl.dataflow.dflow DEBUG: Task 327 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 327 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 327 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 327 has memoization hash 8a9aaf9b709ca99080699a670b6559e3 parsl.dataflow.memoization INFO: Task 327 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb12840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 327 try 0 launched on executor multi with executor id 328 parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/2604833f-a121-49ab-ae73-03440f21a1b3_isrForCrosstalkSources_855820_59.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/2604833f-a121-49ab-ae73-03440f21a1b3_isrForCrosstalkSources_855820_59.stderr parsl.dataflow.dflow DEBUG: Task 328 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 328 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 328 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 328 has memoization hash 87fcc7444a5420ebf9b6705fff446524 parsl.dataflow.memoization INFO: Task 328 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb12980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 328 try 0 launched on executor multi with executor id 329 parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/c4b90971-b144-4d43-b439-b4bc61cc84cf_isrForCrosstalkSources_855910_29.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/c4b90971-b144-4d43-b439-b4bc61cc84cf_isrForCrosstalkSources_855910_29.stderr parsl.dataflow.dflow DEBUG: Task 329 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 329 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 329 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 329 has memoization hash bbc85fc2fb4c0f35e39f13d17f7601ab parsl.dataflow.memoization INFO: Task 329 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb128e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 329 try 0 launched on executor multi with executor id 330 parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/cb46f9d6-88c4-477f-bdbc-b0e912c1a2ee_isrForCrosstalkSources_855909_38.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/cb46f9d6-88c4-477f-bdbc-b0e912c1a2ee_isrForCrosstalkSources_855909_38.stderr parsl.dataflow.dflow DEBUG: Task 330 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 330 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 330 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 330 has memoization hash 0e94b715f44fa6208056422739a18ede parsl.dataflow.memoization INFO: Task 330 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb12a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 330 try 0 launched on executor multi with executor id 331 parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/0983558b-06a1-4785-809b-c9bbc4e392ef_isrForCrosstalkSources_855821_24.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/0983558b-06a1-4785-809b-c9bbc4e392ef_isrForCrosstalkSources_855821_24.stderr parsl.dataflow.dflow DEBUG: Task 331 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 331 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 331 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 331 has memoization hash 56b0f272e216781294ebd31779b328cd parsl.dataflow.memoization INFO: Task 331 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb12ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 331 try 0 launched on executor multi with executor id 332 parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/b679e40d-79e4-4db4-ab4a-4854c758914e_isrForCrosstalkSources_855820_54.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/b679e40d-79e4-4db4-ab4a-4854c758914e_isrForCrosstalkSources_855820_54.stderr parsl.dataflow.dflow DEBUG: Task 332 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 332 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 332 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 332 has memoization hash 255c0022a01b08a688663a28dc0b016a parsl.dataflow.memoization INFO: Task 332 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb12b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 332 try 0 launched on executor multi with executor id 333 parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/72e5b858-e552-466d-86a4-015f6939f399_isrForCrosstalkSources_855822_25.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/72e5b858-e552-466d-86a4-015f6939f399_isrForCrosstalkSources_855822_25.stderr parsl.dataflow.dflow DEBUG: Task 333 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 333 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 333 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 333 has memoization hash 4db25eb0995d6f689eb54cda9d39bbf6 parsl.dataflow.memoization INFO: Task 333 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb12ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 333 try 0 launched on executor multi with executor id 334 parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/3132d585-94a2-4c71-89b1-add95b35a38d_isrForCrosstalkSources_855821_40.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/3132d585-94a2-4c71-89b1-add95b35a38d_isrForCrosstalkSources_855821_40.stderr parsl.dataflow.dflow DEBUG: Task 334 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 334 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 334 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 334 has memoization hash 2ec5cebe69d8b0c80bb5917f62b2f467 parsl.dataflow.memoization INFO: Task 334 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb12c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 334 try 0 launched on executor multi with executor id 335 parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/9c824a62-331f-45fa-8fe0-8a0ddd5c9f2e_isrForCrosstalkSources_855820_43.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/9c824a62-331f-45fa-8fe0-8a0ddd5c9f2e_isrForCrosstalkSources_855820_43.stderr parsl.dataflow.dflow DEBUG: Task 335 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 335 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 335 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 335 has memoization hash d262d45c2443ba035e298f76630acde4 parsl.dataflow.memoization INFO: Task 335 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb12d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 335 try 0 launched on executor multi with executor id 336 parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/1e6b0344-25a8-4fac-ad18-d833931319ca_isrForCrosstalkSources_855911_58.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/1e6b0344-25a8-4fac-ad18-d833931319ca_isrForCrosstalkSources_855911_58.stderr parsl.dataflow.dflow DEBUG: Task 336 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 336 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 336 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 336 has memoization hash e6a1bcf5ad7351dba216258d193ebaf1 parsl.dataflow.memoization INFO: Task 336 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb12de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 336 try 0 launched on executor multi with executor id 337 parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/65085679-9ca8-4b78-9053-3ccca513cb66_isrForCrosstalkSources_855909_35.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/65085679-9ca8-4b78-9053-3ccca513cb66_isrForCrosstalkSources_855909_35.stderr parsl.dataflow.dflow DEBUG: Task 337 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 337 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 337 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 337 has memoization hash 633b694fe653ea1146140ca7c203242e parsl.dataflow.memoization INFO: Task 337 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb12e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 337 try 0 launched on executor multi with executor id 338 parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/c8d7be2d-699a-41e5-a2fa-207eefdcffe6_isrForCrosstalkSources_855911_39.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/c8d7be2d-699a-41e5-a2fa-207eefdcffe6_isrForCrosstalkSources_855911_39.stderr parsl.dataflow.dflow DEBUG: Task 338 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 338 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 338 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 338 has memoization hash 60b49f8e570dc23cf6af7782f0a6df50 parsl.dataflow.memoization INFO: Task 338 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb12fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 338 try 0 launched on executor multi with executor id 339 parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/7544a6e0-4384-4601-b5dc-f7cb019dff1c_isrForCrosstalkSources_855821_8.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/7544a6e0-4384-4601-b5dc-f7cb019dff1c_isrForCrosstalkSources_855821_8.stderr parsl.dataflow.dflow DEBUG: Task 339 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 339 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 339 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 339 has memoization hash 807555cacd8fa55588f25abd8212939f parsl.dataflow.memoization INFO: Task 339 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb12f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 339 try 0 launched on executor multi with executor id 340 parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/f9f30cef-d7ca-4a8c-a181-be8ad0075aaa_isrForCrosstalkSources_855911_19.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/f9f30cef-d7ca-4a8c-a181-be8ad0075aaa_isrForCrosstalkSources_855911_19.stderr parsl.dataflow.dflow DEBUG: Task 340 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 340 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 340 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 340 has memoization hash 3999abb5388952a2ceec50f7bdfbce47 parsl.dataflow.memoization INFO: Task 340 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb13060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 340 try 0 launched on executor multi with executor id 341 parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/5ba133c1-7da1-480a-b0e8-fe2b62927ee5_isrForCrosstalkSources_855820_10.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/5ba133c1-7da1-480a-b0e8-fe2b62927ee5_isrForCrosstalkSources_855820_10.stderr parsl.dataflow.dflow DEBUG: Task 341 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 341 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 341 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 341 has memoization hash b25cf291e0e005cb3c9a75f72c7ccb7e parsl.dataflow.memoization INFO: Task 341 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb13100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 341 try 0 launched on executor multi with executor id 342 parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/caaa3e28-fbd8-4413-8263-57e21b931695_isrForCrosstalkSources_855822_24.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/caaa3e28-fbd8-4413-8263-57e21b931695_isrForCrosstalkSources_855822_24.stderr parsl.dataflow.dflow DEBUG: Task 342 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 342 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 342 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 342 has memoization hash 52d83d12de658fedbfc95a5b86d8092d parsl.dataflow.memoization INFO: Task 342 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb131a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 342 try 0 launched on executor multi with executor id 343 parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/3fa4efd6-50b3-4e2b-bebb-8620739d25fc_isrForCrosstalkSources_855821_5.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/3fa4efd6-50b3-4e2b-bebb-8620739d25fc_isrForCrosstalkSources_855821_5.stderr parsl.dataflow.dflow DEBUG: Task 343 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 343 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 343 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 343 has memoization hash 5cdd30fd04c9daf0fdf60569796f45be parsl.dataflow.memoization INFO: Task 343 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb132e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 343 try 0 launched on executor multi with executor id 344 parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/6955dac6-f691-4a52-aba4-9a3ffaefbcbd_isrForCrosstalkSources_855910_50.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/6955dac6-f691-4a52-aba4-9a3ffaefbcbd_isrForCrosstalkSources_855910_50.stderr parsl.dataflow.dflow DEBUG: Task 344 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 344 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 344 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 344 has memoization hash 89d05f23fe238502a5c5ac30adc096d5 parsl.dataflow.memoization INFO: Task 344 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb13240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 344 try 0 launched on executor multi with executor id 345 parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/452b1d6e-6fa4-4564-8d75-35dd391d0438_isrForCrosstalkSources_855822_27.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/452b1d6e-6fa4-4564-8d75-35dd391d0438_isrForCrosstalkSources_855822_27.stderr parsl.dataflow.dflow DEBUG: Task 345 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 345 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 345 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 345 has memoization hash c773a06779c81bb16e690ad34ef70dcf parsl.dataflow.memoization INFO: Task 345 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb13380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 345 try 0 launched on executor multi with executor id 346 parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/e45b6946-39cd-43ac-b171-cef8296b49d1_isrForCrosstalkSources_855911_55.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/e45b6946-39cd-43ac-b171-cef8296b49d1_isrForCrosstalkSources_855911_55.stderr parsl.dataflow.dflow DEBUG: Task 346 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 346 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 346 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 346 has memoization hash 39ebd695e7410ab377fac3d52a3d1347 parsl.dataflow.memoization INFO: Task 346 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb13420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 346 try 0 launched on executor multi with executor id 347 parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/69d66858-f93e-4a2f-b7f6-2683e82d95da_isrForCrosstalkSources_855820_32.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/69d66858-f93e-4a2f-b7f6-2683e82d95da_isrForCrosstalkSources_855820_32.stderr parsl.dataflow.dflow DEBUG: Task 347 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 347 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 347 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 347 has memoization hash 83b1a59621a8077a72f72dc89d89cc7b parsl.dataflow.memoization INFO: Task 347 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb134c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 347 try 0 launched on executor multi with executor id 348 parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/0f4f52a1-eb9e-4192-b920-a26da581eada_isrForCrosstalkSources_855821_42.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/0f4f52a1-eb9e-4192-b920-a26da581eada_isrForCrosstalkSources_855821_42.stderr parsl.dataflow.dflow DEBUG: Task 348 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 348 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 348 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 348 has memoization hash 60a054ef23811d640702a86915c2a74d parsl.dataflow.memoization INFO: Task 348 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb13600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 348 try 0 launched on executor multi with executor id 349 parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/6214c10f-5f4a-411d-a1d4-2b2284ce4e72_isrForCrosstalkSources_855821_60.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/6214c10f-5f4a-411d-a1d4-2b2284ce4e72_isrForCrosstalkSources_855821_60.stderr parsl.dataflow.dflow DEBUG: Task 349 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 349 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 349 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 349 has memoization hash 2c2568c620e3b6d92b734e7e628d0a3f parsl.dataflow.memoization INFO: Task 349 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb13560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 349 try 0 launched on executor multi with executor id 350 parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/baa7d245-f436-44ab-83f6-ece0d486e9f0_isrForCrosstalkSources_855909_13.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/baa7d245-f436-44ab-83f6-ece0d486e9f0_isrForCrosstalkSources_855909_13.stderr parsl.dataflow.dflow DEBUG: Task 350 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 350 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 350 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 350 has memoization hash 83f882c7cc9944491d0418b58def5bb9 parsl.dataflow.memoization INFO: Task 350 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb136a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 350 try 0 launched on executor multi with executor id 351 parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/b18c4e06-b70b-46ea-9b66-b4dc5ea65000_isrForCrosstalkSources_855822_48.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/b18c4e06-b70b-46ea-9b66-b4dc5ea65000_isrForCrosstalkSources_855822_48.stderr parsl.dataflow.dflow DEBUG: Task 351 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 351 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 351 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 351 has memoization hash 8474f73a01d981622b4b2e2602ca4527 parsl.dataflow.memoization INFO: Task 351 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb13740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 351 try 0 launched on executor multi with executor id 352 parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/4fa3ae39-b7e9-4a0f-9b3f-607ef49e6efc_isrForCrosstalkSources_855910_34.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/4fa3ae39-b7e9-4a0f-9b3f-607ef49e6efc_isrForCrosstalkSources_855910_34.stderr parsl.dataflow.dflow DEBUG: Task 352 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 352 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 352 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 352 has memoization hash 5f1076e35be5d0e785765c0cb1fc159e parsl.dataflow.memoization INFO: Task 352 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb137e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 352 try 0 launched on executor multi with executor id 353 parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/42d37109-06a6-4c1e-9e26-77ebd8f94e8b_isrForCrosstalkSources_855822_19.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/42d37109-06a6-4c1e-9e26-77ebd8f94e8b_isrForCrosstalkSources_855822_19.stderr parsl.dataflow.dflow DEBUG: Task 353 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 353 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 353 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 353 has memoization hash 9dfc7d1b555a461b38739e7259bf3d96 parsl.dataflow.memoization INFO: Task 353 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb13920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 353 try 0 launched on executor multi with executor id 354 parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/6bf1df54-5d7f-476f-a759-9fb6f9e083fb_isrForCrosstalkSources_855910_58.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/6bf1df54-5d7f-476f-a759-9fb6f9e083fb_isrForCrosstalkSources_855910_58.stderr parsl.dataflow.dflow DEBUG: Task 354 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 354 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 354 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 354 has memoization hash 96d44f02d893c625d8e0254b605a3824 parsl.dataflow.memoization INFO: Task 354 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb13880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 354 try 0 launched on executor multi with executor id 355 parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/77aff1c3-68a2-46a7-9208-3f44cd80223a_isrForCrosstalkSources_855822_55.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/77aff1c3-68a2-46a7-9208-3f44cd80223a_isrForCrosstalkSources_855822_55.stderr parsl.dataflow.dflow DEBUG: Task 355 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 355 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 355 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 355 has memoization hash 7d64bb15f9208fb42b43dfec3d5c409f parsl.dataflow.memoization INFO: Task 355 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb139c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 355 try 0 launched on executor multi with executor id 356 parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/ba6e7f2a-e001-46f4-a68e-7225d388c7c7_isrForCrosstalkSources_855821_43.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/ba6e7f2a-e001-46f4-a68e-7225d388c7c7_isrForCrosstalkSources_855821_43.stderr parsl.dataflow.dflow DEBUG: Task 356 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 356 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 356 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 356 has memoization hash d63bd59e7dfb019cd25d2fcd884408bb parsl.dataflow.memoization INFO: Task 356 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb13a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 356 try 0 launched on executor multi with executor id 357 parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/d9cab460-65d9-491e-be97-f15c91123674_isrForCrosstalkSources_855911_29.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/d9cab460-65d9-491e-be97-f15c91123674_isrForCrosstalkSources_855911_29.stderr parsl.dataflow.dflow DEBUG: Task 357 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 357 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 357 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 357 has memoization hash 65fac8d524909fb15519bfe4c5fa9a24 parsl.dataflow.memoization INFO: Task 357 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb13b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 357 try 0 launched on executor multi with executor id 358 parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/b70aefe4-ffbc-492f-92e7-11b92054c5f7_isrForCrosstalkSources_855822_7.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/b70aefe4-ffbc-492f-92e7-11b92054c5f7_isrForCrosstalkSources_855822_7.stderr parsl.dataflow.dflow DEBUG: Task 358 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 358 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 358 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 358 has memoization hash 07af143984c6989eb453de9471bbd443 parsl.dataflow.memoization INFO: Task 358 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb13c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 358 try 0 launched on executor multi with executor id 359 parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/4c20dbe4-8153-4636-a972-f6ca52a94ddc_isrForCrosstalkSources_855821_45.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/4c20dbe4-8153-4636-a972-f6ca52a94ddc_isrForCrosstalkSources_855821_45.stderr parsl.dataflow.dflow DEBUG: Task 359 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 359 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 359 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 359 has memoization hash d82b6e3e9f2a92e561327c48f329062f parsl.dataflow.memoization INFO: Task 359 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb13ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 359 try 0 launched on executor multi with executor id 360 parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/13979976-074d-4b3e-bada-af4b0f8575aa_isrForCrosstalkSources_855820_45.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/13979976-074d-4b3e-bada-af4b0f8575aa_isrForCrosstalkSources_855820_45.stderr parsl.dataflow.dflow DEBUG: Task 360 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 360 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 360 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 360 has memoization hash d257a47e973fc1f057953fd281bd21af parsl.dataflow.memoization INFO: Task 360 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb13ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 360 try 0 launched on executor multi with executor id 361 parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/bda9872a-8638-432c-a37f-68dd76105613_isrForCrosstalkSources_855822_9.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/bda9872a-8638-432c-a37f-68dd76105613_isrForCrosstalkSources_855822_9.stderr parsl.dataflow.dflow DEBUG: Task 361 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 361 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 361 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 361 has memoization hash 0db9bd2e84dcc55178c9bda6fb5c7db9 parsl.dataflow.memoization INFO: Task 361 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb13d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 361 try 0 launched on executor multi with executor id 362 parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/7d92a25b-f402-471d-90dc-dfca83c2ce9c_isrForCrosstalkSources_855822_10.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/7d92a25b-f402-471d-90dc-dfca83c2ce9c_isrForCrosstalkSources_855822_10.stderr parsl.dataflow.dflow DEBUG: Task 362 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 362 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 362 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 362 has memoization hash a033bcdb24f08c085d5fca97b412192e parsl.dataflow.memoization INFO: Task 362 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb13e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 362 try 0 launched on executor multi with executor id 363 parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/7143b02f-f9e6-44e5-b3f2-7236f9726871_isrForCrosstalkSources_855910_19.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/7143b02f-f9e6-44e5-b3f2-7236f9726871_isrForCrosstalkSources_855910_19.stderr parsl.dataflow.dflow DEBUG: Task 363 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 363 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 363 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 363 has memoization hash 572d54305704cb6da33a1204a04323ab parsl.dataflow.memoization INFO: Task 363 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb13f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 363 try 0 launched on executor multi with executor id 364 parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/8bd21cef-1bbf-4b58-b3d8-bc3e66e044d6_isrForCrosstalkSources_855820_55.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/8bd21cef-1bbf-4b58-b3d8-bc3e66e044d6_isrForCrosstalkSources_855820_55.stderr parsl.dataflow.dflow DEBUG: Task 364 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 364 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 364 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 364 has memoization hash d839f63a05dcbf23e7a5f018def9af94 parsl.dataflow.memoization INFO: Task 364 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb13ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 364 try 0 launched on executor multi with executor id 365 parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/4f2c09f2-6021-4e18-9fb6-b07ec33e06e7_isrForCrosstalkSources_855909_57.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/4f2c09f2-6021-4e18-9fb6-b07ec33e06e7_isrForCrosstalkSources_855909_57.stderr parsl.dataflow.dflow DEBUG: Task 365 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 365 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 365 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 365 has memoization hash 6d0473a77821e93833c0631c555f51b2 parsl.dataflow.memoization INFO: Task 365 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dcb100e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 365 try 0 launched on executor multi with executor id 366 parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/0174a0f7-4502-4861-ade3-359921b53cb5_isrForCrosstalkSources_855822_49.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/0174a0f7-4502-4861-ade3-359921b53cb5_isrForCrosstalkSources_855822_49.stderr parsl.dataflow.dflow DEBUG: Task 366 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 366 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 366 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 366 has memoization hash 29dcc586fbd47f0bced4e825093628c5 parsl.dataflow.memoization INFO: Task 366 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dc9a40e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 366 try 0 launched on executor multi with executor id 367 parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/66ac3c44-f514-43c5-8373-06f4e94fb18c_isrForCrosstalkSources_855911_57.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/66ac3c44-f514-43c5-8373-06f4e94fb18c_isrForCrosstalkSources_855911_57.stderr parsl.dataflow.dflow DEBUG: Task 367 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 367 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 367 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 367 has memoization hash c933ca5fa1357f76299e3b628508c837 parsl.dataflow.memoization INFO: Task 367 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dc9a4180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 367 try 0 launched on executor multi with executor id 368 parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/a750bce9-13f4-4b4d-92fb-a6e8168c2540_isrForCrosstalkSources_855910_46.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/a750bce9-13f4-4b4d-92fb-a6e8168c2540_isrForCrosstalkSources_855910_46.stderr parsl.dataflow.dflow DEBUG: Task 368 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 368 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 368 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 368 has memoization hash 28f69254e4d4f4d3445f815abd7df41f parsl.dataflow.memoization INFO: Task 368 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dc9a42c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 368 try 0 launched on executor multi with executor id 369 parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/0b37fdd1-59b1-4bb0-bc1d-149e98b10252_isrForCrosstalkSources_855909_8.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/0b37fdd1-59b1-4bb0-bc1d-149e98b10252_isrForCrosstalkSources_855909_8.stderr parsl.dataflow.dflow DEBUG: Task 369 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 369 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 369 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 369 has memoization hash d753b5243ff7d207129cb46834ffa397 parsl.dataflow.memoization INFO: Task 369 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dc9a4220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 369 try 0 launched on executor multi with executor id 370 parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/eb821e3c-4f02-4ce8-9f2c-1d159bae605d_isrForCrosstalkSources_855820_14.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/eb821e3c-4f02-4ce8-9f2c-1d159bae605d_isrForCrosstalkSources_855820_14.stderr parsl.dataflow.dflow DEBUG: Task 370 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 370 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 370 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 370 has memoization hash 2668468b069cc4ec9d75ec72d68a07e3 parsl.dataflow.memoization INFO: Task 370 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dc9a4360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 370 try 0 launched on executor multi with executor id 371 parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/fbfd239d-40f2-4696-8b81-f66ee81f2af0_isrForCrosstalkSources_855910_13.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/fbfd239d-40f2-4696-8b81-f66ee81f2af0_isrForCrosstalkSources_855910_13.stderr parsl.dataflow.dflow DEBUG: Task 371 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 371 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 371 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 371 has memoization hash aa7ca770f5c706c23391079f041da27f parsl.dataflow.memoization INFO: Task 371 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3dc9a4400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 371 try 0 launched on executor multi with executor id 372 parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/f34e34c9-0ca4-475a-864e-67138acbd4ed_isrForCrosstalkSources_855821_10.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/f34e34c9-0ca4-475a-864e-67138acbd4ed_isrForCrosstalkSources_855821_10.stderr 2024-03-09 20:49:37 proc_lsst.multi:146 [INFO] found job 35332 in provider local proc_lsst.multi INFO: found job 35332 in provider local parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 372 parsl.dataflow.strategy DEBUG: Executor multi has 372 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 92 more blocks parsl.executors.status_handling INFO: Scaling out by 92 blocks parsl.executors.status_handling INFO: Allocated block ID 1 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1 2024-03-09 20:49:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 2024-03-09 20:49:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-09 20:49:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:49:38 proc_lsst.multi:170 [INFO] [multi] job_id 16968750 proc_lsst.multi INFO: [multi] job_id 16968750 2024-03-09 20:49:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:38 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16968750 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968750 parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 16968750 parsl.executors.status_handling INFO: Allocated block ID 2 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2 2024-03-09 20:49:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 2024-03-09 20:49:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:49:38 proc_lsst.multi:170 [INFO] [multi] job_id 16968751 proc_lsst.multi INFO: [multi] job_id 16968751 2024-03-09 20:49:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-09 20:49:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:38 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16968751 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968751 parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 16968751 parsl.executors.status_handling INFO: Allocated block ID 3 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3 2024-03-09 20:49:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 2024-03-09 20:49:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-09 20:49:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:49:38 proc_lsst.multi:170 [INFO] [multi] job_id 16968752 proc_lsst.multi INFO: [multi] job_id 16968752 2024-03-09 20:49:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-09 20:49:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:38 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16968752 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968752 parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 16968752 parsl.executors.status_handling INFO: Allocated block ID 4 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4 2024-03-09 20:49:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 2024-03-09 20:49:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-09 20:49:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:49:39 proc_lsst.multi:170 [INFO] [multi] job_id 16968753 proc_lsst.multi INFO: [multi] job_id 16968753 2024-03-09 20:49:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-09 20:49:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:39 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16968753 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968753 parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 16968753 parsl.executors.status_handling INFO: Allocated block ID 5 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5 2024-03-09 20:49:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 2024-03-09 20:49:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-09 20:49:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:49:39 proc_lsst.multi:170 [INFO] [multi] job_id 16968754 proc_lsst.multi INFO: [multi] job_id 16968754 2024-03-09 20:49:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:49:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:39 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16968754 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968754 parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 16968754 parsl.executors.status_handling INFO: Allocated block ID 6 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6 2024-03-09 20:49:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 2024-03-09 20:49:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:49:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:49:40 proc_lsst.multi:170 [INFO] [multi] job_id 16968755 proc_lsst.multi INFO: [multi] job_id 16968755 2024-03-09 20:49:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-09 20:49:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:40 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16968755 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968755 parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 16968755 parsl.executors.status_handling INFO: Allocated block ID 7 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7 2024-03-09 20:49:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 2024-03-09 20:49:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-09 20:49:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:49:40 proc_lsst.multi:170 [INFO] [multi] job_id 16968756 proc_lsst.multi INFO: [multi] job_id 16968756 2024-03-09 20:49:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-09 20:49:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:40 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16968756 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968756 parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 16968756 parsl.executors.status_handling INFO: Allocated block ID 8 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8 2024-03-09 20:49:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 2024-03-09 20:49:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-09 20:49:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:49:40 proc_lsst.multi:170 [INFO] [multi] job_id 16968757 proc_lsst.multi INFO: [multi] job_id 16968757 2024-03-09 20:49:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-09 20:49:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:40 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16968757 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968757 parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 16968757 parsl.executors.status_handling INFO: Allocated block ID 9 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9 2024-03-09 20:49:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 2024-03-09 20:49:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-09 20:49:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:49:41 proc_lsst.multi:170 [INFO] [multi] job_id 16968758 proc_lsst.multi INFO: [multi] job_id 16968758 2024-03-09 20:49:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-09 20:49:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:41 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16968758 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968758 parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 16968758 parsl.executors.status_handling INFO: Allocated block ID 10 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10 2024-03-09 20:49:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 2024-03-09 20:49:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-09 20:49:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:49:41 proc_lsst.multi:170 [INFO] [multi] job_id 16968759 proc_lsst.multi INFO: [multi] job_id 16968759 2024-03-09 20:49:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-09 20:49:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:41 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16968759 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968759 parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 16968759 parsl.executors.status_handling INFO: Allocated block ID 11 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11 2024-03-09 20:49:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 2024-03-09 20:49:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-09 20:49:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:49:42 proc_lsst.multi:170 [INFO] [multi] job_id 16968760 proc_lsst.multi INFO: [multi] job_id 16968760 2024-03-09 20:49:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-09 20:49:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:42 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16968760 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968760 parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 16968760 parsl.executors.status_handling INFO: Allocated block ID 12 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12 2024-03-09 20:49:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 2024-03-09 20:49:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-09 20:49:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:49:42 proc_lsst.multi:170 [INFO] [multi] job_id 16968761 proc_lsst.multi INFO: [multi] job_id 16968761 2024-03-09 20:49:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-09 20:49:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:42 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16968761 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968761 parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 16968761 parsl.executors.status_handling INFO: Allocated block ID 13 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13 2024-03-09 20:49:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 2024-03-09 20:49:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-09 20:49:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:49:43 proc_lsst.multi:170 [INFO] [multi] job_id 16968762 proc_lsst.multi INFO: [multi] job_id 16968762 2024-03-09 20:49:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-09 20:49:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:43 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16968762 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968762 parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 16968762 parsl.executors.status_handling INFO: Allocated block ID 14 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14 2024-03-09 20:49:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 2024-03-09 20:49:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-09 20:49:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:49:43 proc_lsst.multi:170 [INFO] [multi] job_id 16968763 proc_lsst.multi INFO: [multi] job_id 16968763 2024-03-09 20:49:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-09 20:49:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:43 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16968763 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968763 parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 16968763 parsl.executors.status_handling INFO: Allocated block ID 15 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15 2024-03-09 20:49:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 2024-03-09 20:49:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-09 20:49:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:49:43 proc_lsst.multi:170 [INFO] [multi] job_id 16968764 proc_lsst.multi INFO: [multi] job_id 16968764 2024-03-09 20:49:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-09 20:49:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:43 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16968764 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968764 parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 16968764 parsl.executors.status_handling INFO: Allocated block ID 16 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16 2024-03-09 20:49:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 2024-03-09 20:49:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-09 20:49:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:49:44 proc_lsst.multi:170 [INFO] [multi] job_id 16968765 proc_lsst.multi INFO: [multi] job_id 16968765 2024-03-09 20:49:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-09 20:49:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:44 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16968765 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968765 parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 16968765 parsl.executors.status_handling INFO: Allocated block ID 17 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17 2024-03-09 20:49:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 2024-03-09 20:49:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-09 20:49:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:49:44 proc_lsst.multi:170 [INFO] [multi] job_id 16968766 proc_lsst.multi INFO: [multi] job_id 16968766 2024-03-09 20:49:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-09 20:49:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:44 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16968766 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968766 parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 16968766 parsl.executors.status_handling INFO: Allocated block ID 18 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18 2024-03-09 20:49:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 2024-03-09 20:49:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-09 20:49:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:49:45 proc_lsst.multi:170 [INFO] [multi] job_id 16968767 proc_lsst.multi INFO: [multi] job_id 16968767 2024-03-09 20:49:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-09 20:49:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:45 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16968767 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968767 parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 16968767 parsl.executors.status_handling INFO: Allocated block ID 19 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19 2024-03-09 20:49:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 2024-03-09 20:49:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-09 20:49:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:49:46 proc_lsst.multi:170 [INFO] [multi] job_id 16968768 proc_lsst.multi INFO: [multi] job_id 16968768 2024-03-09 20:49:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-09 20:49:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:46 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16968768 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968768 parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 16968768 parsl.executors.status_handling INFO: Allocated block ID 20 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20 2024-03-09 20:49:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 2024-03-09 20:49:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-09 20:49:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:49:46 proc_lsst.multi:170 [INFO] [multi] job_id 16968769 proc_lsst.multi INFO: [multi] job_id 16968769 2024-03-09 20:49:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 20:49:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:46 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16968769 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968769 parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 16968769 parsl.executors.status_handling INFO: Allocated block ID 21 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21 2024-03-09 20:49:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 2024-03-09 20:49:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 20:49:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:49:46 proc_lsst.multi:170 [INFO] [multi] job_id 16968770 proc_lsst.multi INFO: [multi] job_id 16968770 2024-03-09 20:49:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-09 20:49:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:46 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16968770 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968770 parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 16968770 parsl.executors.status_handling INFO: Allocated block ID 22 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22 2024-03-09 20:49:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 2024-03-09 20:49:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-09 20:49:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:49:47 proc_lsst.multi:170 [INFO] [multi] job_id 16968771 proc_lsst.multi INFO: [multi] job_id 16968771 2024-03-09 20:49:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-09 20:49:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:47 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16968771 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968771 parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 16968771 parsl.executors.status_handling INFO: Allocated block ID 23 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23 2024-03-09 20:49:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 2024-03-09 20:49:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-09 20:49:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:49:47 proc_lsst.multi:170 [INFO] [multi] job_id 16968772 proc_lsst.multi INFO: [multi] job_id 16968772 2024-03-09 20:49:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-09 20:49:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:47 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16968772 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968772 parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 16968772 parsl.executors.status_handling INFO: Allocated block ID 24 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24 2024-03-09 20:49:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 2024-03-09 20:49:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-09 20:49:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:49:48 proc_lsst.multi:170 [INFO] [multi] job_id 16968773 proc_lsst.multi INFO: [multi] job_id 16968773 2024-03-09 20:49:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-09 20:49:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:48 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16968773 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968773 parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 16968773 parsl.executors.status_handling INFO: Allocated block ID 25 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25 2024-03-09 20:49:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 2024-03-09 20:49:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-09 20:49:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:49:48 proc_lsst.multi:170 [INFO] [multi] job_id 16968774 proc_lsst.multi INFO: [multi] job_id 16968774 2024-03-09 20:49:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-09 20:49:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:48 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16968774 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968774 parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 16968774 parsl.executors.status_handling INFO: Allocated block ID 26 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26 2024-03-09 20:49:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 2024-03-09 20:49:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-09 20:49:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:49:49 proc_lsst.multi:170 [INFO] [multi] job_id 16968775 proc_lsst.multi INFO: [multi] job_id 16968775 2024-03-09 20:49:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-09 20:49:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:49 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16968775 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968775 parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 16968775 parsl.executors.status_handling INFO: Allocated block ID 27 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27 2024-03-09 20:49:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 2024-03-09 20:49:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-09 20:49:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:49:49 proc_lsst.multi:170 [INFO] [multi] job_id 16968776 proc_lsst.multi INFO: [multi] job_id 16968776 2024-03-09 20:49:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-09 20:49:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:49 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16968776 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968776 parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 16968776 parsl.executors.status_handling INFO: Allocated block ID 28 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28 2024-03-09 20:49:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 2024-03-09 20:49:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-09 20:49:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:49:50 proc_lsst.multi:170 [INFO] [multi] job_id 16968777 proc_lsst.multi INFO: [multi] job_id 16968777 2024-03-09 20:49:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-09 20:49:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:50 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16968777 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968777 parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 16968777 parsl.executors.status_handling INFO: Allocated block ID 29 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29 2024-03-09 20:49:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 2024-03-09 20:49:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-09 20:49:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:49:50 proc_lsst.multi:170 [INFO] [multi] job_id 16968778 proc_lsst.multi INFO: [multi] job_id 16968778 2024-03-09 20:49:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-09 20:49:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:50 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16968778 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968778 parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 16968778 parsl.executors.status_handling INFO: Allocated block ID 30 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30 2024-03-09 20:49:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 2024-03-09 20:49:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-09 20:49:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:49:51 proc_lsst.multi:170 [INFO] [multi] job_id 16968779 proc_lsst.multi INFO: [multi] job_id 16968779 2024-03-09 20:49:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:49:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:51 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16968779 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16968779 parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 16968779 parsl.executors.status_handling INFO: Allocated block ID 31 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-31 2024-03-09 20:49:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 2024-03-09 20:49:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:49:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:51 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-09 20:49:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:49:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23c1b35578eaab278162ebb1e53fde7d with result from task 1 parsl.dataflow.dflow DEBUG: Task 1 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/ebe1783d-e57c-4518-aaa3-9a1b903e8f26_isrForCrosstalkSources_855909_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/ebe1783d-e57c-4518-aaa3-9a1b903e8f26_isrForCrosstalkSources_855909_25.stderr parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9730308ce7b3fd1d9b8a1a0d8c453774 with result from task 0 parsl.dataflow.dflow DEBUG: Task 0 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/bfc14203-d22c-46d9-bf29-c6a347b53121_isrForCrosstalkSources_855820_51.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/bfc14203-d22c-46d9-bf29-c6a347b53121_isrForCrosstalkSources_855820_51.stderr 2024-03-09 20:49:53 proc_lsst.multi:170 [INFO] [multi] job_id 16968780 proc_lsst.multi INFO: [multi] job_id 16968780 2024-03-09 20:49:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:49:53 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16968780 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16968780 parsl.executors.status_handling DEBUG: Launched block 31 on executor multi with job ID 16968780 parsl.executors.status_handling INFO: Allocated block ID 32 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-32 2024-03-09 20:49:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 2024-03-09 20:49:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:49:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:53 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:49:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:49:54 proc_lsst.multi:170 [INFO] [multi] job_id 16968781 proc_lsst.multi INFO: [multi] job_id 16968781 2024-03-09 20:49:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-09 20:49:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:49:54 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16968781 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16968781 parsl.executors.status_handling DEBUG: Launched block 32 on executor multi with job ID 16968781 parsl.executors.status_handling INFO: Allocated block ID 33 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-33 2024-03-09 20:49:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 2024-03-09 20:49:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:49:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:54 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-09 20:49:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:49:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:49:54 proc_lsst.multi:170 [INFO] [multi] job_id 16968782 proc_lsst.multi INFO: [multi] job_id 16968782 2024-03-09 20:49:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-09 20:49:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:49:54 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16968782 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16968782 parsl.executors.status_handling DEBUG: Launched block 33 on executor multi with job ID 16968782 parsl.executors.status_handling INFO: Allocated block ID 34 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-34 2024-03-09 20:49:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 2024-03-09 20:49:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:49:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:54 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-09 20:49:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:49:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:49:55 proc_lsst.multi:170 [INFO] [multi] job_id 16968783 proc_lsst.multi INFO: [multi] job_id 16968783 2024-03-09 20:49:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-09 20:49:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:49:55 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16968783 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16968783 parsl.executors.status_handling DEBUG: Launched block 34 on executor multi with job ID 16968783 parsl.executors.status_handling INFO: Allocated block ID 35 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-35 2024-03-09 20:49:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 2024-03-09 20:49:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:49:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:55 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-09 20:49:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:49:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:49:55 proc_lsst.multi:170 [INFO] [multi] job_id 16968784 proc_lsst.multi INFO: [multi] job_id 16968784 2024-03-09 20:49:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:49:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:49:55 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16968784 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16968784 parsl.executors.status_handling DEBUG: Launched block 35 on executor multi with job ID 16968784 parsl.executors.status_handling INFO: Allocated block ID 36 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-36 2024-03-09 20:49:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 2024-03-09 20:49:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:49:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:55 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:49:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:49:55 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-09 20:49:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:49:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:49:56 proc_lsst.multi:170 [INFO] [multi] job_id 16968785 proc_lsst.multi INFO: [multi] job_id 16968785 2024-03-09 20:49:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:49:56 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16968785 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16968785 parsl.executors.status_handling DEBUG: Launched block 36 on executor multi with job ID 16968785 parsl.executors.status_handling INFO: Allocated block ID 37 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-37 2024-03-09 20:49:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 2024-03-09 20:49:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:49:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:49:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:49:56 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:49:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:49:56 proc_lsst.multi:170 [INFO] [multi] job_id 16968786 proc_lsst.multi INFO: [multi] job_id 16968786 2024-03-09 20:49:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-09 20:49:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:49:56 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16968786 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16968786 parsl.executors.status_handling DEBUG: Launched block 37 on executor multi with job ID 16968786 parsl.executors.status_handling INFO: Allocated block ID 38 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-38 2024-03-09 20:49:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 2024-03-09 20:49:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:49:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:49:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:49:56 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-09 20:49:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:49:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:49:57 proc_lsst.multi:170 [INFO] [multi] job_id 16968787 proc_lsst.multi INFO: [multi] job_id 16968787 2024-03-09 20:49:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-09 20:49:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:49:57 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16968787 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16968787 parsl.executors.status_handling DEBUG: Launched block 38 on executor multi with job ID 16968787 parsl.executors.status_handling INFO: Allocated block ID 39 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-39 2024-03-09 20:49:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 2024-03-09 20:49:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:49:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:57 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:49:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:49:57 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-09 20:49:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:49:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:49:58 proc_lsst.multi:170 [INFO] [multi] job_id 16968788 proc_lsst.multi INFO: [multi] job_id 16968788 2024-03-09 20:49:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-09 20:49:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:49:58 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16968788 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16968788 parsl.executors.status_handling DEBUG: Launched block 39 on executor multi with job ID 16968788 parsl.executors.status_handling INFO: Allocated block ID 40 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-40 2024-03-09 20:49:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 2024-03-09 20:49:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:49:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:58 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:49:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:49:58 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-09 20:49:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:49:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:49:58 proc_lsst.multi:170 [INFO] [multi] job_id 16968789 proc_lsst.multi INFO: [multi] job_id 16968789 2024-03-09 20:49:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:49:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:49:58 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16968789 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16968789 parsl.executors.status_handling DEBUG: Launched block 40 on executor multi with job ID 16968789 parsl.executors.status_handling INFO: Allocated block ID 41 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-41 2024-03-09 20:49:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 2024-03-09 20:49:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:49:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:58 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:49:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:49:58 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:49:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:49:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:49:58 proc_lsst.multi:170 [INFO] [multi] job_id 16968790 proc_lsst.multi INFO: [multi] job_id 16968790 2024-03-09 20:49:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-09 20:49:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:49:58 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16968790 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16968790 parsl.executors.status_handling DEBUG: Launched block 41 on executor multi with job ID 16968790 parsl.executors.status_handling INFO: Allocated block ID 42 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-42 2024-03-09 20:49:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 2024-03-09 20:49:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:49:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:58 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:49:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:49:58 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-09 20:49:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:49:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:49:58 proc_lsst.multi:170 [INFO] [multi] job_id 16968791 proc_lsst.multi INFO: [multi] job_id 16968791 2024-03-09 20:49:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-09 20:49:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:49:58 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16968791 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16968791 parsl.executors.status_handling DEBUG: Launched block 42 on executor multi with job ID 16968791 parsl.executors.status_handling INFO: Allocated block ID 43 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-43 2024-03-09 20:49:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 2024-03-09 20:49:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:49:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:58 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:49:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:49:58 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-09 20:49:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:49:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:49:59 proc_lsst.multi:170 [INFO] [multi] job_id 16968792 proc_lsst.multi INFO: [multi] job_id 16968792 2024-03-09 20:49:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-09 20:49:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:49:59 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16968792 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16968792 parsl.executors.status_handling DEBUG: Launched block 43 on executor multi with job ID 16968792 parsl.executors.status_handling INFO: Allocated block ID 44 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-44 2024-03-09 20:49:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 2024-03-09 20:49:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:49:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:59 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:49:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:49:59 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-09 20:49:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:49:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:49:59 proc_lsst.multi:170 [INFO] [multi] job_id 16968793 proc_lsst.multi INFO: [multi] job_id 16968793 2024-03-09 20:49:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-09 20:49:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:49:59 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16968793 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16968793 parsl.executors.status_handling DEBUG: Launched block 44 on executor multi with job ID 16968793 parsl.executors.status_handling INFO: Allocated block ID 45 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-45 2024-03-09 20:49:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 2024-03-09 20:49:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:49:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:59 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:49:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:49:59 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-09 20:49:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:49:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca3347482424541120dec6dd8eeb2115 with result from task 2 parsl.dataflow.dflow DEBUG: Task 2 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/c858dc63-c41e-4b23-acc7-af08f3000ede_isrForCrosstalkSources_855821_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/c858dc63-c41e-4b23-acc7-af08f3000ede_isrForCrosstalkSources_855821_49.stderr 2024-03-09 20:49:59 proc_lsst.multi:170 [INFO] [multi] job_id 16968794 proc_lsst.multi INFO: [multi] job_id 16968794 2024-03-09 20:49:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-09 20:49:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:49:59 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16968794 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16968794 parsl.executors.status_handling DEBUG: Launched block 45 on executor multi with job ID 16968794 parsl.executors.status_handling INFO: Allocated block ID 46 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-46 2024-03-09 20:49:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 2024-03-09 20:49:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:49:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:59 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:49:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:49:59 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-09 20:49:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:49:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:49:59 proc_lsst.multi:170 [INFO] [multi] job_id 16968795 proc_lsst.multi INFO: [multi] job_id 16968795 2024-03-09 20:49:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-09 20:49:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:49:59 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16968795 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16968795 parsl.executors.status_handling DEBUG: Launched block 46 on executor multi with job ID 16968795 parsl.executors.status_handling INFO: Allocated block ID 47 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-47 2024-03-09 20:49:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 2024-03-09 20:49:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:49:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:49:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:49:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:49:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:49:59 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:49:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:49:59 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:49:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-09 20:49:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:49:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:50:00 proc_lsst.multi:170 [INFO] [multi] job_id 16968796 proc_lsst.multi INFO: [multi] job_id 16968796 2024-03-09 20:50:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-09 20:50:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:00 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16968796 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16968796 parsl.executors.status_handling DEBUG: Launched block 47 on executor multi with job ID 16968796 parsl.executors.status_handling INFO: Allocated block ID 48 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-48 2024-03-09 20:50:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 2024-03-09 20:50:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:00 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:00 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-09 20:50:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:50:00 proc_lsst.multi:170 [INFO] [multi] job_id 16968797 proc_lsst.multi INFO: [multi] job_id 16968797 2024-03-09 20:50:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-09 20:50:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:00 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16968797 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16968797 parsl.executors.status_handling DEBUG: Launched block 48 on executor multi with job ID 16968797 parsl.executors.status_handling INFO: Allocated block ID 49 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-49 2024-03-09 20:50:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 2024-03-09 20:50:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:00 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:00 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-09 20:50:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:50:00 proc_lsst.multi:170 [INFO] [multi] job_id 16968798 proc_lsst.multi INFO: [multi] job_id 16968798 2024-03-09 20:50:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-09 20:50:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:00 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16968798 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16968798 parsl.executors.status_handling DEBUG: Launched block 49 on executor multi with job ID 16968798 parsl.executors.status_handling INFO: Allocated block ID 50 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-50 2024-03-09 20:50:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 2024-03-09 20:50:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:00 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:00 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-09 20:50:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:50:00 proc_lsst.multi:170 [INFO] [multi] job_id 16968799 proc_lsst.multi INFO: [multi] job_id 16968799 2024-03-09 20:50:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-09 20:50:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:00 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16968799 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16968799 parsl.executors.status_handling DEBUG: Launched block 50 on executor multi with job ID 16968799 parsl.executors.status_handling INFO: Allocated block ID 51 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-51 2024-03-09 20:50:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 2024-03-09 20:50:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:00 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:00 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-09 20:50:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:50:01 proc_lsst.multi:170 [INFO] [multi] job_id 16968800 proc_lsst.multi INFO: [multi] job_id 16968800 2024-03-09 20:50:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-09 20:50:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:01 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16968800 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16968800 parsl.executors.status_handling DEBUG: Launched block 51 on executor multi with job ID 16968800 parsl.executors.status_handling INFO: Allocated block ID 52 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-52 2024-03-09 20:50:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 2024-03-09 20:50:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:01 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:01 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-09 20:50:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:50:01 proc_lsst.multi:170 [INFO] [multi] job_id 16968805 proc_lsst.multi INFO: [multi] job_id 16968805 2024-03-09 20:50:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-09 20:50:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:01 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16968805 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16968805 parsl.executors.status_handling DEBUG: Launched block 52 on executor multi with job ID 16968805 parsl.executors.status_handling INFO: Allocated block ID 53 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-53 2024-03-09 20:50:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 2024-03-09 20:50:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:01 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:01 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-09 20:50:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:50:02 proc_lsst.multi:170 [INFO] [multi] job_id 16968811 proc_lsst.multi INFO: [multi] job_id 16968811 2024-03-09 20:50:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-09 20:50:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:02 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16968811 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16968811 parsl.executors.status_handling DEBUG: Launched block 53 on executor multi with job ID 16968811 parsl.executors.status_handling INFO: Allocated block ID 54 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-54 2024-03-09 20:50:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 2024-03-09 20:50:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:02 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:02 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-09 20:50:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:50:02 proc_lsst.multi:170 [INFO] [multi] job_id 16968812 proc_lsst.multi INFO: [multi] job_id 16968812 2024-03-09 20:50:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-09 20:50:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:02 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16968812 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16968812 parsl.executors.status_handling DEBUG: Launched block 54 on executor multi with job ID 16968812 parsl.executors.status_handling INFO: Allocated block ID 55 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-55 2024-03-09 20:50:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 2024-03-09 20:50:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:02 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:02 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-09 20:50:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:50:03 proc_lsst.multi:170 [INFO] [multi] job_id 16968813 proc_lsst.multi INFO: [multi] job_id 16968813 2024-03-09 20:50:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 20:50:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:03 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16968813 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16968813 parsl.executors.status_handling DEBUG: Launched block 55 on executor multi with job ID 16968813 parsl.executors.status_handling INFO: Allocated block ID 56 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-56 2024-03-09 20:50:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 2024-03-09 20:50:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:03 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:03 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 20:50:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:03 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-09 20:50:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:50:03 proc_lsst.multi:170 [INFO] [multi] job_id 16968814 proc_lsst.multi INFO: [multi] job_id 16968814 2024-03-09 20:50:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:03 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16968814 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968814 parsl.executors.status_handling DEBUG: Launched block 56 on executor multi with job ID 16968814 parsl.executors.status_handling INFO: Allocated block ID 57 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-57 2024-03-09 20:50:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 2024-03-09 20:50:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:03 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:03 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 20:50:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:03 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:50:04 proc_lsst.multi:170 [INFO] [multi] job_id 16968815 proc_lsst.multi INFO: [multi] job_id 16968815 2024-03-09 20:50:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-09 20:50:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:04 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16968815 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968815 parsl.executors.status_handling DEBUG: Launched block 57 on executor multi with job ID 16968815 parsl.executors.status_handling INFO: Allocated block ID 58 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-58 2024-03-09 20:50:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 2024-03-09 20:50:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:04 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:04 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 20:50:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:04 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-09 20:50:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6242cb7c13f59e88578c9f03737a0619 with result from task 3 parsl.dataflow.dflow DEBUG: Task 3 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/7163b6fe-9d83-4149-a098-4330052fbdc6_isrForCrosstalkSources_855820_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/7163b6fe-9d83-4149-a098-4330052fbdc6_isrForCrosstalkSources_855820_53.stderr 2024-03-09 20:50:04 proc_lsst.multi:170 [INFO] [multi] job_id 16968816 proc_lsst.multi INFO: [multi] job_id 16968816 2024-03-09 20:50:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-09 20:50:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:04 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16968816 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968816 parsl.executors.status_handling DEBUG: Launched block 58 on executor multi with job ID 16968816 parsl.executors.status_handling INFO: Allocated block ID 59 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-59 2024-03-09 20:50:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 2024-03-09 20:50:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:04 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:04 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 20:50:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:04 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-09 20:50:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 4 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dfbc2c30ce8604a5515ae1a8f5753216 with result from task 4 parsl.dataflow.dflow DEBUG: Task 4 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/e47bdb68-8e57-40ed-9d4b-bd93f061788e_isrForCrosstalkSources_855911_14.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/e47bdb68-8e57-40ed-9d4b-bd93f061788e_isrForCrosstalkSources_855911_14.stderr 2024-03-09 20:50:05 proc_lsst.multi:170 [INFO] [multi] job_id 16968817 proc_lsst.multi INFO: [multi] job_id 16968817 2024-03-09 20:50:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-09 20:50:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:05 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16968817 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968817 parsl.executors.status_handling DEBUG: Launched block 59 on executor multi with job ID 16968817 parsl.executors.status_handling INFO: Allocated block ID 60 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-60 2024-03-09 20:50:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 2024-03-09 20:50:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:05 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:05 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 20:50:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:05 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-09 20:50:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 5 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb58569d2ec10e135d7742fba5f2cd08 with result from task 5 parsl.dataflow.dflow DEBUG: Task 5 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/629bff43-6703-4a2c-bcb0-18e4dc9236b7_isrForCrosstalkSources_855911_52.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/629bff43-6703-4a2c-bcb0-18e4dc9236b7_isrForCrosstalkSources_855911_52.stderr 2024-03-09 20:50:05 proc_lsst.multi:170 [INFO] [multi] job_id 16968818 proc_lsst.multi INFO: [multi] job_id 16968818 2024-03-09 20:50:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:05 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16968818 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968818 parsl.executors.status_handling DEBUG: Launched block 60 on executor multi with job ID 16968818 parsl.executors.status_handling INFO: Allocated block ID 61 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-61 2024-03-09 20:50:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 2024-03-09 20:50:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:05 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:05 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 20:50:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:05 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:50:06 proc_lsst.multi:170 [INFO] [multi] job_id 16968819 proc_lsst.multi INFO: [multi] job_id 16968819 2024-03-09 20:50:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-09 20:50:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:06 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16968819 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968819 parsl.executors.status_handling DEBUG: Launched block 61 on executor multi with job ID 16968819 parsl.executors.status_handling INFO: Allocated block ID 62 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-62 2024-03-09 20:50:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-62 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-62 2024-03-09 20:50:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:06 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:06 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 20:50:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:06 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-09 20:50:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:50:06 proc_lsst.multi:170 [INFO] [multi] job_id 16968823 proc_lsst.multi INFO: [multi] job_id 16968823 2024-03-09 20:50:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-09 20:50:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:06 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16968823 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968823 parsl.executors.status_handling DEBUG: Launched block 62 on executor multi with job ID 16968823 parsl.executors.status_handling INFO: Allocated block ID 63 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-63 2024-03-09 20:50:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-63 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-63 2024-03-09 20:50:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:06 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:06 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 20:50:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:06 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-09 20:50:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d86293cc63517cb6161795e2e7db631 with result from task 6 parsl.dataflow.dflow DEBUG: Task 6 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/52dc29e6-dbd9-426f-b32f-36a66300c4a4_isrForCrosstalkSources_855910_62.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/52dc29e6-dbd9-426f-b32f-36a66300c4a4_isrForCrosstalkSources_855910_62.stderr 2024-03-09 20:50:07 proc_lsst.multi:170 [INFO] [multi] job_id 16968824 proc_lsst.multi INFO: [multi] job_id 16968824 2024-03-09 20:50:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-09 20:50:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:07 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16968824 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968824 parsl.executors.status_handling DEBUG: Launched block 63 on executor multi with job ID 16968824 parsl.executors.status_handling INFO: Allocated block ID 64 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-64 2024-03-09 20:50:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-64 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-64 2024-03-09 20:50:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:07 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:07 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 20:50:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:07 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-09 20:50:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:50:07 proc_lsst.multi:170 [INFO] [multi] job_id 16968825 proc_lsst.multi INFO: [multi] job_id 16968825 2024-03-09 20:50:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-09 20:50:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:07 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16968825 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968825 parsl.executors.status_handling DEBUG: Launched block 64 on executor multi with job ID 16968825 parsl.executors.status_handling INFO: Allocated block ID 65 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-65 2024-03-09 20:50:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-65 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-65 2024-03-09 20:50:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:07 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:07 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 20:50:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:07 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-09 20:50:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:50:08 proc_lsst.multi:170 [INFO] [multi] job_id 16968826 proc_lsst.multi INFO: [multi] job_id 16968826 2024-03-09 20:50:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-09 20:50:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:08 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16968826 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968826 parsl.executors.status_handling DEBUG: Launched block 65 on executor multi with job ID 16968826 parsl.executors.status_handling INFO: Allocated block ID 66 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-66 2024-03-09 20:50:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-66 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-66 2024-03-09 20:50:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:08 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:08 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 20:50:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:08 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-09 20:50:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:50:08 proc_lsst.multi:170 [INFO] [multi] job_id 16968827 proc_lsst.multi INFO: [multi] job_id 16968827 2024-03-09 20:50:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-09 20:50:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:08 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16968827 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968827 parsl.executors.status_handling DEBUG: Launched block 66 on executor multi with job ID 16968827 parsl.executors.status_handling INFO: Allocated block ID 67 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-67 2024-03-09 20:50:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-67 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-67 2024-03-09 20:50:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:08 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:08 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 20:50:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:08 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-09 20:50:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:50:09 proc_lsst.multi:170 [INFO] [multi] job_id 16968828 proc_lsst.multi INFO: [multi] job_id 16968828 2024-03-09 20:50:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-09 20:50:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:09 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16968828 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968828 parsl.executors.status_handling DEBUG: Launched block 67 on executor multi with job ID 16968828 parsl.executors.status_handling INFO: Allocated block ID 68 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-68 2024-03-09 20:50:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-68 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-68 2024-03-09 20:50:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:09 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:09 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 20:50:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:09 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-09 20:50:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:50:09 proc_lsst.multi:170 [INFO] [multi] job_id 16968829 proc_lsst.multi INFO: [multi] job_id 16968829 2024-03-09 20:50:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-09 20:50:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:09 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16968829 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968829 parsl.executors.status_handling DEBUG: Launched block 68 on executor multi with job ID 16968829 parsl.executors.status_handling INFO: Allocated block ID 69 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-69 2024-03-09 20:50:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-69 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-69 2024-03-09 20:50:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:09 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:09 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 20:50:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:09 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-09 20:50:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:50:10 proc_lsst.multi:170 [INFO] [multi] job_id 16968830 proc_lsst.multi INFO: [multi] job_id 16968830 2024-03-09 20:50:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-09 20:50:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:10 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16968830 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968830 parsl.executors.status_handling DEBUG: Launched block 69 on executor multi with job ID 16968830 parsl.executors.status_handling INFO: Allocated block ID 70 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-70 2024-03-09 20:50:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-70 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-70 2024-03-09 20:50:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:10 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:10 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 20:50:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:10 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-09 20:50:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:50:10 proc_lsst.multi:170 [INFO] [multi] job_id 16968831 proc_lsst.multi INFO: [multi] job_id 16968831 2024-03-09 20:50:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-09 20:50:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:10 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16968831 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968831 parsl.executors.status_handling DEBUG: Launched block 70 on executor multi with job ID 16968831 parsl.executors.status_handling INFO: Allocated block ID 71 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-71 2024-03-09 20:50:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-71 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-71 2024-03-09 20:50:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:10 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:10 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 20:50:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:10 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-09 20:50:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:50:11 proc_lsst.multi:170 [INFO] [multi] job_id 16968832 proc_lsst.multi INFO: [multi] job_id 16968832 2024-03-09 20:50:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-09 20:50:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:11 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16968832 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968832 parsl.executors.status_handling DEBUG: Launched block 71 on executor multi with job ID 16968832 parsl.executors.status_handling INFO: Allocated block ID 72 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-72 2024-03-09 20:50:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-72 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-72 2024-03-09 20:50:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:11 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:11 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 20:50:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:11 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-09 20:50:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:50:12 proc_lsst.multi:170 [INFO] [multi] job_id 16968833 proc_lsst.multi INFO: [multi] job_id 16968833 2024-03-09 20:50:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-09 20:50:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:12 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16968833 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968833 parsl.executors.status_handling DEBUG: Launched block 72 on executor multi with job ID 16968833 parsl.executors.status_handling INFO: Allocated block ID 73 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-73 2024-03-09 20:50:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-73 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-73 2024-03-09 20:50:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:12 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:12 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 20:50:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:12 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-09 20:50:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:50:12 proc_lsst.multi:170 [INFO] [multi] job_id 16968834 proc_lsst.multi INFO: [multi] job_id 16968834 2024-03-09 20:50:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-09 20:50:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:12 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16968834 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968834 parsl.executors.status_handling DEBUG: Launched block 73 on executor multi with job ID 16968834 parsl.executors.status_handling INFO: Allocated block ID 74 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-74 2024-03-09 20:50:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-74 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-74 2024-03-09 20:50:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:12 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:12 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 20:50:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:12 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-09 20:50:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:50:12 proc_lsst.multi:170 [INFO] [multi] job_id 16968835 proc_lsst.multi INFO: [multi] job_id 16968835 2024-03-09 20:50:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-09 20:50:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:12 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16968835 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968835 parsl.executors.status_handling DEBUG: Launched block 74 on executor multi with job ID 16968835 parsl.executors.status_handling INFO: Allocated block ID 75 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-75 2024-03-09 20:50:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-75 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-75 2024-03-09 20:50:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:12 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:12 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 20:50:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:12 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-09 20:50:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:50:13 proc_lsst.multi:170 [INFO] [multi] job_id 16968836 proc_lsst.multi INFO: [multi] job_id 16968836 2024-03-09 20:50:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 20:50:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:13 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16968836 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968836 parsl.executors.status_handling DEBUG: Launched block 75 on executor multi with job ID 16968836 parsl.executors.status_handling INFO: Allocated block ID 76 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-76 2024-03-09 20:50:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-76 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-76 2024-03-09 20:50:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:13 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:13 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 20:50:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:13 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 20:50:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:50:13 proc_lsst.multi:170 [INFO] [multi] job_id 16968837 proc_lsst.multi INFO: [multi] job_id 16968837 2024-03-09 20:50:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-09 20:50:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:13 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16968837 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968837 parsl.executors.status_handling DEBUG: Launched block 76 on executor multi with job ID 16968837 parsl.executors.status_handling INFO: Allocated block ID 77 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-77 2024-03-09 20:50:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-77 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-77 2024-03-09 20:50:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:13 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:13 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 20:50:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:13 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-09 20:50:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 34 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3466ec710de251ef5645b608c8548b2 with result from task 34 parsl.dataflow.dflow DEBUG: Task 34 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/529dfbe1-963d-44c0-b886-7f39d26d5a9a_isrForCrosstalkSources_855909_2.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/529dfbe1-963d-44c0-b886-7f39d26d5a9a_isrForCrosstalkSources_855909_2.stderr 2024-03-09 20:50:13 proc_lsst.multi:170 [INFO] [multi] job_id 16968838 proc_lsst.multi INFO: [multi] job_id 16968838 2024-03-09 20:50:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-09 20:50:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:13 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16968838 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968838 parsl.executors.status_handling DEBUG: Launched block 77 on executor multi with job ID 16968838 parsl.executors.status_handling INFO: Allocated block ID 78 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-78 2024-03-09 20:50:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-78 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-78 2024-03-09 20:50:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:13 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:13 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 20:50:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:13 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-09 20:50:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:50:14 proc_lsst.multi:170 [INFO] [multi] job_id 16968839 proc_lsst.multi INFO: [multi] job_id 16968839 2024-03-09 20:50:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-09 20:50:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:14 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16968839 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968839 parsl.executors.status_handling DEBUG: Launched block 78 on executor multi with job ID 16968839 parsl.executors.status_handling INFO: Allocated block ID 79 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-79 2024-03-09 20:50:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-79 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-79 2024-03-09 20:50:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:14 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:14 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 20:50:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:14 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-09 20:50:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:50:14 proc_lsst.multi:170 [INFO] [multi] job_id 16968840 proc_lsst.multi INFO: [multi] job_id 16968840 2024-03-09 20:50:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-09 20:50:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:14 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16968840 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968840 parsl.executors.status_handling DEBUG: Launched block 79 on executor multi with job ID 16968840 parsl.executors.status_handling INFO: Allocated block ID 80 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-80 2024-03-09 20:50:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-80 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-80 2024-03-09 20:50:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:14 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:14 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 20:50:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:14 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-09 20:50:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:50:14 proc_lsst.multi:170 [INFO] [multi] job_id 16968841 proc_lsst.multi INFO: [multi] job_id 16968841 2024-03-09 20:50:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-09 20:50:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:14 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16968841 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968841 parsl.executors.status_handling DEBUG: Launched block 80 on executor multi with job ID 16968841 parsl.executors.status_handling INFO: Allocated block ID 81 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-81 2024-03-09 20:50:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-81 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-81 2024-03-09 20:50:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:14 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:14 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 20:50:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:14 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-09 20:50:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:50:14 proc_lsst.multi:170 [INFO] [multi] job_id 16968842 proc_lsst.multi INFO: [multi] job_id 16968842 2024-03-09 20:50:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-09 20:50:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:14 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16968842 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968842 parsl.executors.status_handling DEBUG: Launched block 81 on executor multi with job ID 16968842 parsl.executors.status_handling INFO: Allocated block ID 82 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-82 2024-03-09 20:50:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-82 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-82 2024-03-09 20:50:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:14 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:14 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 20:50:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:14 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-09 20:50:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:50:15 proc_lsst.multi:170 [INFO] [multi] job_id 16968843 proc_lsst.multi INFO: [multi] job_id 16968843 2024-03-09 20:50:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-09 20:50:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:15 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16968843 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968843 parsl.executors.status_handling DEBUG: Launched block 82 on executor multi with job ID 16968843 parsl.executors.status_handling INFO: Allocated block ID 83 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-83 2024-03-09 20:50:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-83 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-83 2024-03-09 20:50:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:15 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 20:50:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:15 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-09 20:50:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:50:15 proc_lsst.multi:170 [INFO] [multi] job_id 16968844 proc_lsst.multi INFO: [multi] job_id 16968844 2024-03-09 20:50:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-09 20:50:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:15 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16968844 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968844 parsl.executors.status_handling DEBUG: Launched block 83 on executor multi with job ID 16968844 parsl.executors.status_handling INFO: Allocated block ID 84 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-84 2024-03-09 20:50:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-84 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-84 2024-03-09 20:50:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:15 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 20:50:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:15 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-09 20:50:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:50:16 proc_lsst.multi:170 [INFO] [multi] job_id 16968845 proc_lsst.multi INFO: [multi] job_id 16968845 2024-03-09 20:50:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-09 20:50:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:16 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16968845 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968845 parsl.executors.status_handling DEBUG: Launched block 84 on executor multi with job ID 16968845 parsl.executors.status_handling INFO: Allocated block ID 85 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-85 2024-03-09 20:50:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-85 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-85 2024-03-09 20:50:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 20:50:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:16 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-09 20:50:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:50:16 proc_lsst.multi:170 [INFO] [multi] job_id 16968846 proc_lsst.multi INFO: [multi] job_id 16968846 2024-03-09 20:50:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:16 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16968846 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968846 parsl.executors.status_handling DEBUG: Launched block 85 on executor multi with job ID 16968846 parsl.executors.status_handling INFO: Allocated block ID 86 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-86 2024-03-09 20:50:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-86 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-86 2024-03-09 20:50:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 20:50:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:16 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:50:17 proc_lsst.multi:170 [INFO] [multi] job_id 16968847 proc_lsst.multi INFO: [multi] job_id 16968847 2024-03-09 20:50:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 31 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 31 2024-03-09 20:50:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:17 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16968847 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968847 parsl.executors.status_handling DEBUG: Launched block 86 on executor multi with job ID 16968847 parsl.executors.status_handling INFO: Allocated block ID 87 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-87 2024-03-09 20:50:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-87 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-87 2024-03-09 20:50:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:17 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:17 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 20:50:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:17 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 31 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 31 2024-03-09 20:50:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:50:17 proc_lsst.multi:170 [INFO] [multi] job_id 16968848 proc_lsst.multi INFO: [multi] job_id 16968848 2024-03-09 20:50:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 32 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 32 2024-03-09 20:50:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:17 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16968848 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968848 parsl.executors.status_handling DEBUG: Launched block 87 on executor multi with job ID 16968848 parsl.executors.status_handling INFO: Allocated block ID 88 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-88 2024-03-09 20:50:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-88 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-88 2024-03-09 20:50:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:17 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:17 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 20:50:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:17 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 32 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 32 2024-03-09 20:50:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:50:18 proc_lsst.multi:170 [INFO] [multi] job_id 16968849 proc_lsst.multi INFO: [multi] job_id 16968849 2024-03-09 20:50:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 33 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 33 2024-03-09 20:50:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:18 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16968849 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968849 parsl.executors.status_handling DEBUG: Launched block 88 on executor multi with job ID 16968849 parsl.executors.status_handling INFO: Allocated block ID 89 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-89 2024-03-09 20:50:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-89 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-89 2024-03-09 20:50:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:18 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:18 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 20:50:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:18 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 33 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 33 2024-03-09 20:50:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 37 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d9f6f99b8108271c22fd8f7bebece5c with result from task 37 parsl.dataflow.dflow DEBUG: Task 37 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/9eba03a8-b3fb-40fe-8d94-d29bbf9cb8a6_isrForCrosstalkSources_855910_60.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/9eba03a8-b3fb-40fe-8d94-d29bbf9cb8a6_isrForCrosstalkSources_855910_60.stderr 2024-03-09 20:50:18 proc_lsst.multi:170 [INFO] [multi] job_id 16968850 proc_lsst.multi INFO: [multi] job_id 16968850 2024-03-09 20:50:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 34 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 34 2024-03-09 20:50:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:18 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16968850 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968850 parsl.executors.status_handling DEBUG: Launched block 89 on executor multi with job ID 16968850 parsl.executors.status_handling INFO: Allocated block ID 90 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-90 2024-03-09 20:50:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-90 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-90 2024-03-09 20:50:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:18 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:18 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 20:50:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:18 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 34 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 34 2024-03-09 20:50:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:50:19 proc_lsst.multi:170 [INFO] [multi] job_id 16968851 proc_lsst.multi INFO: [multi] job_id 16968851 2024-03-09 20:50:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 35 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 35 2024-03-09 20:50:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:19 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16968851 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968851 parsl.executors.status_handling DEBUG: Launched block 90 on executor multi with job ID 16968851 parsl.executors.status_handling INFO: Allocated block ID 91 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-91 2024-03-09 20:50:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-91 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-91 2024-03-09 20:50:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:19 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 20:50:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:19 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 35 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 35 2024-03-09 20:50:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:50:19 proc_lsst.multi:170 [INFO] [multi] job_id 16968852 proc_lsst.multi INFO: [multi] job_id 16968852 2024-03-09 20:50:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 36 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 36 2024-03-09 20:50:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:19 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16968852 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968852 parsl.executors.status_handling DEBUG: Launched block 91 on executor multi with job ID 16968852 parsl.executors.status_handling INFO: Allocated block ID 92 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-92 2024-03-09 20:50:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-92 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-92 2024-03-09 20:50:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-09 20:50:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-09 20:50:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-09 20:50:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-09 20:50:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-09 20:50:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-09 20:50:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-09 20:50:19 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-09 20:50:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-09 20:50:19 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-09 20:50:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 36 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 36 2024-03-09 20:50:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,10.64.65.10,198.48.92.26,169.254.95.120,127.0.0.1,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54763 --result_port=54243 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-09 20:50:19 proc_lsst.multi:170 [INFO] [multi] job_id 16968853 proc_lsst.multi INFO: [multi] job_id 16968853 2024-03-09 20:50:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 37 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 37 2024-03-09 20:50:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-09 20:50:19 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16968853 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16968853 parsl.executors.status_handling DEBUG: Launched block 92 on executor multi with job ID 16968853 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22762785682704 2024-03-09 20:50:19 proc_lsst.multi:146 [INFO] found job 35332 in provider local proc_lsst.multi INFO: found job 35332 in provider local 2024-03-09 20:50:20 proc_lsst.multi:146 [INFO] found job 16968750 in provider astro proc_lsst.multi INFO: found job 16968750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:20 proc_lsst.multi:146 [INFO] found job 16968751 in provider astro proc_lsst.multi INFO: found job 16968751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 385f7f558af3f8269f48516396ec317c with result from task 36 parsl.dataflow.dflow DEBUG: Task 36 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/d5aa8fd0-79f8-49b1-9da9-e1f8120d32f6_isrForCrosstalkSources_855910_59.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/d5aa8fd0-79f8-49b1-9da9-e1f8120d32f6_isrForCrosstalkSources_855910_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:21 proc_lsst.multi:146 [INFO] found job 16968752 in provider astro proc_lsst.multi INFO: found job 16968752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:21 proc_lsst.multi:146 [INFO] found job 16968753 in provider astro proc_lsst.multi INFO: found job 16968753 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.dataflow.dflow INFO: Task 38 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af27a55e03fc324ce04031ae5e05d726 with result from task 38 parsl.dataflow.dflow DEBUG: Task 38 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/1094d02b-afab-41cc-a7cb-29248d53425a_isrForCrosstalkSources_855822_59.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/1094d02b-afab-41cc-a7cb-29248d53425a_isrForCrosstalkSources_855822_59.stderr parsl.dataflow.dflow INFO: Task 35 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28646f34abf5723fc45f23d26ca46815 with result from task 35 parsl.dataflow.dflow DEBUG: Task 35 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/833a5e7d-bb7f-4654-b3f2-534e2cb0f5f2_isrForCrosstalkSources_855821_13.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/833a5e7d-bb7f-4654-b3f2-534e2cb0f5f2_isrForCrosstalkSources_855821_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:21 proc_lsst.multi:146 [INFO] found job 16968754 in provider astro proc_lsst.multi INFO: found job 16968754 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:22 proc_lsst.multi:146 [INFO] found job 16968755 in provider astro proc_lsst.multi INFO: found job 16968755 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:22 proc_lsst.multi:146 [INFO] found job 16968756 in provider astro proc_lsst.multi INFO: found job 16968756 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:23 proc_lsst.multi:146 [INFO] found job 16968757 in provider astro proc_lsst.multi INFO: found job 16968757 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:23 proc_lsst.multi:146 [INFO] found job 16968758 in provider astro proc_lsst.multi INFO: found job 16968758 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:24 proc_lsst.multi:146 [INFO] found job 16968759 in provider astro proc_lsst.multi INFO: found job 16968759 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.dataflow.dflow INFO: Task 56 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4431a88b1f340d8c4dd0e6dbd1e80cab with result from task 56 parsl.dataflow.dflow DEBUG: Task 56 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/3ae32828-cdee-4d68-b1b0-c4add05e55c8_isrForCrosstalkSources_855820_49.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/3ae32828-cdee-4d68-b1b0-c4add05e55c8_isrForCrosstalkSources_855820_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:26 proc_lsst.multi:146 [INFO] found job 16968760 in provider astro proc_lsst.multi INFO: found job 16968760 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:27 proc_lsst.multi:146 [INFO] found job 16968761 in provider astro proc_lsst.multi INFO: found job 16968761 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:28 proc_lsst.multi:146 [INFO] found job 16968762 in provider astro proc_lsst.multi INFO: found job 16968762 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:28 proc_lsst.multi:146 [INFO] found job 16968763 in provider astro proc_lsst.multi INFO: found job 16968763 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:29 proc_lsst.multi:146 [INFO] found job 16968764 in provider astro proc_lsst.multi INFO: found job 16968764 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:29 proc_lsst.multi:146 [INFO] found job 16968765 in provider astro proc_lsst.multi INFO: found job 16968765 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:30 proc_lsst.multi:146 [INFO] found job 16968766 in provider astro proc_lsst.multi INFO: found job 16968766 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:31 proc_lsst.multi:146 [INFO] found job 16968767 in provider astro proc_lsst.multi INFO: found job 16968767 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:31 proc_lsst.multi:146 [INFO] found job 16968768 in provider astro proc_lsst.multi INFO: found job 16968768 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:32 proc_lsst.multi:146 [INFO] found job 16968769 in provider astro proc_lsst.multi INFO: found job 16968769 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.dataflow.dflow INFO: Task 64 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 376b5bdb6545df8444a3f0542837f9bd with result from task 64 parsl.dataflow.dflow DEBUG: Task 64 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/8920c2bc-e237-45b2-a575-ce1b1060cb9d_isrForCrosstalkSources_855911_45.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/8920c2bc-e237-45b2-a575-ce1b1060cb9d_isrForCrosstalkSources_855911_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:32 proc_lsst.multi:146 [INFO] found job 16968770 in provider astro proc_lsst.multi INFO: found job 16968770 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:33 proc_lsst.multi:146 [INFO] found job 16968771 in provider astro proc_lsst.multi INFO: found job 16968771 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:33 proc_lsst.multi:146 [INFO] found job 16968772 in provider astro proc_lsst.multi INFO: found job 16968772 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:34 proc_lsst.multi:146 [INFO] found job 16968773 in provider astro proc_lsst.multi INFO: found job 16968773 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.dataflow.dflow INFO: Task 66 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e80ffd0e10754062bb8631cc7b75e5c1 with result from task 66 parsl.dataflow.dflow DEBUG: Task 66 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/7e1085fa-a473-4cdc-897c-22c6d723aa59_isrForCrosstalkSources_855911_11.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/7e1085fa-a473-4cdc-897c-22c6d723aa59_isrForCrosstalkSources_855911_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:34 proc_lsst.multi:146 [INFO] found job 16968774 in provider astro proc_lsst.multi INFO: found job 16968774 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:34 proc_lsst.multi:146 [INFO] found job 16968775 in provider astro proc_lsst.multi INFO: found job 16968775 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:35 proc_lsst.multi:146 [INFO] found job 16968776 in provider astro proc_lsst.multi INFO: found job 16968776 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.dataflow.dflow INFO: Task 65 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d60b087b5b5f3d953eba9de87a195711 with result from task 65 parsl.dataflow.dflow DEBUG: Task 65 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/131f4075-4513-4fe6-af03-0a542bed621e_isrForCrosstalkSources_855911_33.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/131f4075-4513-4fe6-af03-0a542bed621e_isrForCrosstalkSources_855911_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:35 proc_lsst.multi:146 [INFO] found job 16968777 in provider astro proc_lsst.multi INFO: found job 16968777 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:36 proc_lsst.multi:146 [INFO] found job 16968778 in provider astro proc_lsst.multi INFO: found job 16968778 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:36 proc_lsst.multi:146 [INFO] found job 16968779 in provider astro proc_lsst.multi INFO: found job 16968779 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:37 proc_lsst.multi:146 [INFO] found job 16968780 in provider escience_rtx6k proc_lsst.multi INFO: found job 16968780 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968780,16968781,16968782,16968783,16968784' parsl.dataflow.dflow INFO: Task 68 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d08f2c354c7bbc1f726617e5f7124de1 with result from task 68 parsl.dataflow.dflow DEBUG: Task 68 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/78895cae-6bff-426a-8e83-5a4dfabfa5f2_isrForCrosstalkSources_855911_16.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/78895cae-6bff-426a-8e83-5a4dfabfa5f2_isrForCrosstalkSources_855911_16.stderr parsl.dataflow.dflow INFO: Task 67 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4bbe324d76c278d004364795b83cc3ca with result from task 67 parsl.dataflow.dflow DEBUG: Task 67 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/b9e67966-1e72-42d1-97f8-6ffc03b8e2a4_isrForCrosstalkSources_855911_42.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/b9e67966-1e72-42d1-97f8-6ffc03b8e2a4_isrForCrosstalkSources_855911_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968781 R 16968782 R 16968783 R 16968784 R 16968780 R parsl.providers.slurm.slurm DEBUG: Updating job 16968781 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968780 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:37 proc_lsst.multi:146 [INFO] found job 16968781 in provider escience_rtx6k proc_lsst.multi INFO: found job 16968781 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968780,16968781,16968782,16968783,16968784' parsl.providers.slurm.slurm DEBUG: squeue returned 16968781 R 16968782 R 16968783 R 16968784 R 16968780 R parsl.providers.slurm.slurm DEBUG: Updating job 16968781 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968780 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:38 proc_lsst.multi:146 [INFO] found job 16968782 in provider escience_rtx6k proc_lsst.multi INFO: found job 16968782 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968780,16968781,16968782,16968783,16968784' parsl.providers.slurm.slurm DEBUG: squeue returned 16968781 R 16968782 R 16968783 R 16968784 R 16968780 R parsl.providers.slurm.slurm DEBUG: Updating job 16968781 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968780 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:38 proc_lsst.multi:146 [INFO] found job 16968783 in provider escience_rtx6k proc_lsst.multi INFO: found job 16968783 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968780,16968781,16968782,16968783,16968784' parsl.providers.slurm.slurm DEBUG: squeue returned 16968781 R 16968782 R 16968783 R 16968784 R 16968780 R parsl.providers.slurm.slurm DEBUG: Updating job 16968781 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968780 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:39 proc_lsst.multi:146 [INFO] found job 16968784 in provider escience_rtx6k proc_lsst.multi INFO: found job 16968784 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968780,16968781,16968782,16968783,16968784' parsl.providers.slurm.slurm DEBUG: squeue returned 16968781 R 16968782 R 16968783 R 16968784 R 16968780 R parsl.providers.slurm.slurm DEBUG: Updating job 16968781 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968780 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:40 proc_lsst.multi:146 [INFO] found job 16968785 in provider escience_a40 proc_lsst.multi INFO: found job 16968785 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:40 proc_lsst.multi:146 [INFO] found job 16968786 in provider escience_a40 proc_lsst.multi INFO: found job 16968786 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:41 proc_lsst.multi:146 [INFO] found job 16968787 in provider escience_a40 proc_lsst.multi INFO: found job 16968787 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:41 proc_lsst.multi:146 [INFO] found job 16968788 in provider escience_a40 proc_lsst.multi INFO: found job 16968788 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:41 proc_lsst.multi:146 [INFO] found job 16968789 in provider escience_a40 proc_lsst.multi INFO: found job 16968789 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:42 proc_lsst.multi:146 [INFO] found job 16968790 in provider escience_a40 proc_lsst.multi INFO: found job 16968790 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.dataflow.dflow INFO: Task 40 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f84a128407667e6d7b447511420d0507 with result from task 40 parsl.dataflow.dflow DEBUG: Task 40 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/70b25e75-a62b-482d-aa80-7e4f80e6d73c_isrForCrosstalkSources_855821_11.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/70b25e75-a62b-482d-aa80-7e4f80e6d73c_isrForCrosstalkSources_855821_11.stderr parsl.dataflow.dflow INFO: Task 39 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13a993332896179fd63f60d8833fdba2 with result from task 39 parsl.dataflow.dflow DEBUG: Task 39 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/02abf915-1523-4e31-b275-b6af998a9f36_isrForCrosstalkSources_855911_26.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/02abf915-1523-4e31-b275-b6af998a9f36_isrForCrosstalkSources_855911_26.stderr parsl.dataflow.dflow INFO: Task 43 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbc6c36c106f7de821fe7f3a11d91084 with result from task 43 parsl.dataflow.dflow DEBUG: Task 43 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/1bd881f6-c1fd-49fb-b6fb-8c238b0ee0ee_isrForCrosstalkSources_855911_15.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/1bd881f6-c1fd-49fb-b6fb-8c238b0ee0ee_isrForCrosstalkSources_855911_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:43 proc_lsst.multi:146 [INFO] found job 16968791 in provider escience_a40 proc_lsst.multi INFO: found job 16968791 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.dataflow.dflow INFO: Task 42 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e707037e3b43820aea58e5afa1c387d with result from task 42 parsl.dataflow.dflow DEBUG: Task 42 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/5560782f-47d9-4559-80d8-5c9c55dbbdef_isrForCrosstalkSources_855909_40.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/5560782f-47d9-4559-80d8-5c9c55dbbdef_isrForCrosstalkSources_855909_40.stderr parsl.dataflow.dflow INFO: Task 41 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1505663db626bc79340ba206e4c23b9d with result from task 41 parsl.dataflow.dflow DEBUG: Task 41 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/9acadd6d-dfe8-4cb4-9735-41ae9a0612fc_isrForCrosstalkSources_855820_23.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/9acadd6d-dfe8-4cb4-9735-41ae9a0612fc_isrForCrosstalkSources_855820_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:43 proc_lsst.multi:146 [INFO] found job 16968792 in provider escience_a40 proc_lsst.multi INFO: found job 16968792 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:44 proc_lsst.multi:146 [INFO] found job 16968793 in provider escience_a40 proc_lsst.multi INFO: found job 16968793 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:44 proc_lsst.multi:146 [INFO] found job 16968794 in provider escience_a40 proc_lsst.multi INFO: found job 16968794 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:45 proc_lsst.multi:146 [INFO] found job 16968795 in provider escience_a40 proc_lsst.multi INFO: found job 16968795 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:45 proc_lsst.multi:146 [INFO] found job 16968796 in provider escience_a40 proc_lsst.multi INFO: found job 16968796 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:45 proc_lsst.multi:146 [INFO] found job 16968797 in provider escience_a40 proc_lsst.multi INFO: found job 16968797 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:46 proc_lsst.multi:146 [INFO] found job 16968798 in provider escience_a40 proc_lsst.multi INFO: found job 16968798 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.dataflow.dflow INFO: Task 70 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 415390b2b8fee6d4310a0982af1d8444 with result from task 70 parsl.dataflow.dflow DEBUG: Task 70 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/597e24b3-d429-4bcb-9659-eb7f9c7ba2f5_isrForCrosstalkSources_855820_56.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/597e24b3-d429-4bcb-9659-eb7f9c7ba2f5_isrForCrosstalkSources_855820_56.stderr parsl.dataflow.dflow INFO: Task 10 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d89c5e967d77bc7b06e10bf5f89b34d1 with result from task 10 parsl.dataflow.dflow DEBUG: Task 10 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/75bb30b4-1d37-4269-8645-1e6d97cce4d7_isrForCrosstalkSources_855821_15.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/75bb30b4-1d37-4269-8645-1e6d97cce4d7_isrForCrosstalkSources_855821_15.stderr parsl.dataflow.dflow INFO: Task 20 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99f8fb6a49f033ad134ee81d357de2fc with result from task 20 parsl.dataflow.dflow DEBUG: Task 20 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/f46496b6-2ebe-438d-8ea7-4eecdedec3cd_isrForCrosstalkSources_855910_45.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/f46496b6-2ebe-438d-8ea7-4eecdedec3cd_isrForCrosstalkSources_855910_45.stderr parsl.dataflow.dflow INFO: Task 31 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d81cea9eda84a0ba83ac0db5bc746b37 with result from task 31 parsl.dataflow.dflow DEBUG: Task 31 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/edde4809-693c-4bbc-8485-2b898f9b828c_isrForCrosstalkSources_855820_6.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/edde4809-693c-4bbc-8485-2b898f9b828c_isrForCrosstalkSources_855820_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:46 proc_lsst.multi:146 [INFO] found job 16968799 in provider escience_a40 proc_lsst.multi INFO: found job 16968799 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5680d40fea48b98731d3445801ffdbc3 with result from task 30 parsl.dataflow.dflow DEBUG: Task 30 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/44c2a1b8-3d1a-4acd-a3c2-006c83c82a25_isrForCrosstalkSources_855911_10.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/44c2a1b8-3d1a-4acd-a3c2-006c83c82a25_isrForCrosstalkSources_855911_10.stderr parsl.dataflow.dflow INFO: Task 19 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94849db646166a04a70626d85ebf7ac9 with result from task 19 parsl.dataflow.dflow DEBUG: Task 19 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/7f3413a8-6f44-41df-97bb-f77f8d03d958_isrForCrosstalkSources_855821_33.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/7f3413a8-6f44-41df-97bb-f77f8d03d958_isrForCrosstalkSources_855821_33.stderr parsl.dataflow.dflow INFO: Task 28 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51f8abf8b955e7f3338c8f92b5f06d4a with result from task 28 parsl.dataflow.dflow DEBUG: Task 28 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/d152e696-0526-469b-bc08-9367bc01738e_isrForCrosstalkSources_855909_59.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/d152e696-0526-469b-bc08-9367bc01738e_isrForCrosstalkSources_855909_59.stderr parsl.dataflow.dflow INFO: Task 11 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 036cf9bd681733b8b49113807ba9bcd6 with result from task 11 parsl.dataflow.dflow DEBUG: Task 11 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/98ccd585-bded-47d6-8dee-4dc6c7095969_isrForCrosstalkSources_855822_61.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/98ccd585-bded-47d6-8dee-4dc6c7095969_isrForCrosstalkSources_855822_61.stderr parsl.dataflow.dflow INFO: Task 17 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6da75f7f4513e224ede3ae9f84c3102 with result from task 17 parsl.dataflow.dflow DEBUG: Task 17 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/6f9b40db-1d33-4c2b-8bf9-c3e0670a16dd_isrForCrosstalkSources_855822_50.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/6f9b40db-1d33-4c2b-8bf9-c3e0670a16dd_isrForCrosstalkSources_855822_50.stderr parsl.dataflow.dflow INFO: Task 23 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 912c09d72f49ca21451454aa9adef074 with result from task 23 parsl.dataflow.dflow DEBUG: Task 23 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/c4ce1a06-2b31-4a8c-8339-d5c6114593fc_isrForCrosstalkSources_855821_35.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/c4ce1a06-2b31-4a8c-8339-d5c6114593fc_isrForCrosstalkSources_855821_35.stderr parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8bc15986828408dcd816487ab51db7d8 with result from task 14 parsl.dataflow.dflow DEBUG: Task 14 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/7f5c489b-702e-45be-9b11-83592dd1223a_isrForCrosstalkSources_855821_32.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/7f5c489b-702e-45be-9b11-83592dd1223a_isrForCrosstalkSources_855821_32.stderr parsl.dataflow.dflow INFO: Task 7 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d2cc8e21270e09daaeda34b74d92a80 with result from task 7 parsl.dataflow.dflow DEBUG: Task 7 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/4323d15f-d656-4997-a51f-1077a1434b6a_isrForCrosstalkSources_855909_16.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/4323d15f-d656-4997-a51f-1077a1434b6a_isrForCrosstalkSources_855909_16.stderr parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f639c0f7268e9728f72df3099962ead with result from task 33 parsl.dataflow.dflow DEBUG: Task 33 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/47625e6d-01c1-41c9-986d-324e9dac5252_isrForCrosstalkSources_855910_54.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/47625e6d-01c1-41c9-986d-324e9dac5252_isrForCrosstalkSources_855910_54.stderr parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 959040d87b41e5c35a8047c9906a0be0 with result from task 15 parsl.dataflow.dflow DEBUG: Task 15 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/80c5f238-c4b4-4429-a37a-070721ae86e3_isrForCrosstalkSources_855909_32.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/80c5f238-c4b4-4429-a37a-070721ae86e3_isrForCrosstalkSources_855909_32.stderr parsl.dataflow.dflow INFO: Task 16 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c309d6dc04bdaf8774df3254de1e5e20 with result from task 16 parsl.dataflow.dflow DEBUG: Task 16 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/11127bbd-bf4e-49ae-b993-3e06f77162d7_isrForCrosstalkSources_855909_7.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/11127bbd-bf4e-49ae-b993-3e06f77162d7_isrForCrosstalkSources_855909_7.stderr parsl.dataflow.dflow INFO: Task 26 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df82657af979c3406712f920a7746eea with result from task 26 parsl.dataflow.dflow DEBUG: Task 26 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/7d181afa-c2fd-458c-af9b-4a318478141a_isrForCrosstalkSources_855820_34.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/7d181afa-c2fd-458c-af9b-4a318478141a_isrForCrosstalkSources_855820_34.stderr parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8264e3d221f66310ad6b11250f5d1b6 with result from task 24 parsl.dataflow.dflow DEBUG: Task 24 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/639ad00f-743d-4cc6-a43d-5b8e5c267243_isrForCrosstalkSources_855910_5.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/639ad00f-743d-4cc6-a43d-5b8e5c267243_isrForCrosstalkSources_855910_5.stderr parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c4ed15b3d5ad4a562647043d43932f3 with result from task 27 parsl.dataflow.dflow DEBUG: Task 27 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/a5753eda-1820-47e4-9316-66eedbb7b152_isrForCrosstalkSources_855910_2.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/a5753eda-1820-47e4-9316-66eedbb7b152_isrForCrosstalkSources_855910_2.stderr parsl.dataflow.dflow INFO: Task 22 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ecc38ec7eb2da95c4e07faa561e3e944 with result from task 22 parsl.dataflow.dflow DEBUG: Task 22 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/3f0fc7fd-e47c-4837-86a2-68a4058c16bd_isrForCrosstalkSources_855822_54.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/3f0fc7fd-e47c-4837-86a2-68a4058c16bd_isrForCrosstalkSources_855822_54.stderr parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc0c8e359f9ba53d71038675109e8255 with result from task 18 parsl.dataflow.dflow DEBUG: Task 18 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/9d5feb91-1328-4bd0-a1b5-5e068259b1bd_isrForCrosstalkSources_855910_9.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/9d5feb91-1328-4bd0-a1b5-5e068259b1bd_isrForCrosstalkSources_855910_9.stderr parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac6f9f25b009df11f168f3e162b7eaca with result from task 9 parsl.dataflow.dflow DEBUG: Task 9 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/b3b73630-8bf7-4e1d-a834-6a3742312723_isrForCrosstalkSources_855821_1.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/b3b73630-8bf7-4e1d-a834-6a3742312723_isrForCrosstalkSources_855821_1.stderr parsl.dataflow.dflow INFO: Task 32 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91130eb60f71de87c949f7e2a600293c with result from task 32 parsl.dataflow.dflow DEBUG: Task 32 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/6462b175-d74b-47a7-a294-a00ef52d2355_isrForCrosstalkSources_855820_25.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/6462b175-d74b-47a7-a294-a00ef52d2355_isrForCrosstalkSources_855820_25.stderr parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5aac479bcb78ee241995d93af9a7e39 with result from task 13 parsl.dataflow.dflow DEBUG: Task 13 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/e8ef6668-3629-47b8-a379-d49c8a229a4d_isrForCrosstalkSources_855821_38.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/e8ef6668-3629-47b8-a379-d49c8a229a4d_isrForCrosstalkSources_855821_38.stderr parsl.dataflow.dflow INFO: Task 25 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2ef8b79c8ea3648fd020b4d93f38b2e with result from task 25 parsl.dataflow.dflow DEBUG: Task 25 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/9924596f-d526-4763-ac2d-b385a6d93d59_isrForCrosstalkSources_855820_21.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/9924596f-d526-4763-ac2d-b385a6d93d59_isrForCrosstalkSources_855820_21.stderr parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84b03dedba2486c71bae636256bd15fa with result from task 12 parsl.dataflow.dflow DEBUG: Task 12 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/298a1c83-1958-433b-8a7b-bcf31af54664_isrForCrosstalkSources_855910_8.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/298a1c83-1958-433b-8a7b-bcf31af54664_isrForCrosstalkSources_855910_8.stderr parsl.dataflow.dflow INFO: Task 8 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6bf303c81925fe2248459f0b57a4ea81 with result from task 8 parsl.dataflow.dflow DEBUG: Task 8 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/7b3be9a2-165e-4dce-91bb-f952df3371a5_isrForCrosstalkSources_855911_40.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/7b3be9a2-165e-4dce-91bb-f952df3371a5_isrForCrosstalkSources_855911_40.stderr parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0e283f00f22fb203b6ebf316a51554f with result from task 21 parsl.dataflow.dflow DEBUG: Task 21 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/410cbe8d-858c-4405-8128-d33cec152849_isrForCrosstalkSources_855910_16.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/410cbe8d-858c-4405-8128-d33cec152849_isrForCrosstalkSources_855910_16.stderr parsl.dataflow.dflow INFO: Task 29 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4614c01a213acdfa0ebd1b11a94e9640 with result from task 29 parsl.dataflow.dflow DEBUG: Task 29 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/4eca3972-92d8-47bd-bc84-7a4368944aa0_isrForCrosstalkSources_855910_23.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/4eca3972-92d8-47bd-bc84-7a4368944aa0_isrForCrosstalkSources_855910_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:46 proc_lsst.multi:146 [INFO] found job 16968800 in provider escience_a40 proc_lsst.multi INFO: found job 16968800 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.dataflow.dflow INFO: Task 58 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d67780a64de113b54a34821b018586a with result from task 58 parsl.dataflow.dflow DEBUG: Task 58 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/4224aa25-7978-4647-955b-dd07f371b6e1_isrForCrosstalkSources_855820_7.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/4224aa25-7978-4647-955b-dd07f371b6e1_isrForCrosstalkSources_855820_7.stderr parsl.dataflow.dflow INFO: Task 55 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 202e76aca9b27d00bf4c350413cbe509 with result from task 55 parsl.dataflow.dflow DEBUG: Task 55 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/a996961c-3b7c-450c-aefd-a2e9694bc088_isrForCrosstalkSources_855822_21.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/a996961c-3b7c-450c-aefd-a2e9694bc088_isrForCrosstalkSources_855822_21.stderr parsl.dataflow.dflow INFO: Task 57 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35608d00d936501bd90932f76ff7f69b with result from task 57 parsl.dataflow.dflow DEBUG: Task 57 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/1148a7d7-e1d9-4b3f-8f51-290ade855724_isrForCrosstalkSources_855822_39.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/1148a7d7-e1d9-4b3f-8f51-290ade855724_isrForCrosstalkSources_855822_39.stderr parsl.dataflow.dflow INFO: Task 63 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 768283743806c2612ced52142bb4f83b with result from task 63 parsl.dataflow.dflow DEBUG: Task 63 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/b642830d-983c-4910-a214-ca1aa987509f_isrForCrosstalkSources_855820_62.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/b642830d-983c-4910-a214-ca1aa987509f_isrForCrosstalkSources_855820_62.stderr parsl.dataflow.dflow INFO: Task 54 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84fec48006d241a2e940ed4d8e96bf46 with result from task 54 parsl.dataflow.dflow DEBUG: Task 54 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/e8b6ca86-849a-4c05-86bd-9e21963a50a5_isrForCrosstalkSources_855911_60.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/e8b6ca86-849a-4c05-86bd-9e21963a50a5_isrForCrosstalkSources_855911_60.stderr parsl.dataflow.dflow INFO: Task 62 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa6234532135f4d74db6729771036892 with result from task 62 parsl.dataflow.dflow DEBUG: Task 62 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/b2edaffd-33ea-4401-ba65-20c9e33d7113_isrForCrosstalkSources_855911_61.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/b2edaffd-33ea-4401-ba65-20c9e33d7113_isrForCrosstalkSources_855911_61.stderr parsl.dataflow.dflow INFO: Task 61 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b978f558c971748ff6121f0c030ef347 with result from task 61 parsl.dataflow.dflow DEBUG: Task 61 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/f59fc9f3-c272-4eb1-af0e-05f629fa1b76_isrForCrosstalkSources_855911_46.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/f59fc9f3-c272-4eb1-af0e-05f629fa1b76_isrForCrosstalkSources_855911_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:47 proc_lsst.multi:146 [INFO] found job 16968805 in provider escience_a40 proc_lsst.multi INFO: found job 16968805 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:47 proc_lsst.multi:146 [INFO] found job 16968811 in provider escience_a40 proc_lsst.multi INFO: found job 16968811 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:47 proc_lsst.multi:146 [INFO] found job 16968812 in provider escience_a40 proc_lsst.multi INFO: found job 16968812 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:48 proc_lsst.multi:146 [INFO] found job 16968813 in provider escience_a40 proc_lsst.multi INFO: found job 16968813 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:48 proc_lsst.multi:146 [INFO] found job 16968814 in provider checkpoint proc_lsst.multi INFO: found job 16968814 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.dataflow.dflow INFO: Task 44 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40fbcc6b5b421f240b52913f18a84fb5 with result from task 44 parsl.dataflow.dflow DEBUG: Task 44 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/0e70ddd2-f6f0-4b3c-a962-0201690d3ef0_isrForCrosstalkSources_855821_26.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/0e70ddd2-f6f0-4b3c-a962-0201690d3ef0_isrForCrosstalkSources_855821_26.stderr parsl.dataflow.dflow INFO: Task 45 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e15c09b6ca71f06070982502641ba778 with result from task 45 parsl.dataflow.dflow DEBUG: Task 45 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/1fd3dcf8-34a9-4753-a5fb-ae5ab6d8b634_isrForCrosstalkSources_855822_12.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/1fd3dcf8-34a9-4753-a5fb-ae5ab6d8b634_isrForCrosstalkSources_855822_12.stderr parsl.dataflow.dflow INFO: Task 46 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24a453709e9bdd3abfe45ea541f531ec with result from task 46 parsl.dataflow.dflow DEBUG: Task 46 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/9008735a-ebdb-48da-9ade-c983df0319e1_isrForCrosstalkSources_855820_41.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/9008735a-ebdb-48da-9ade-c983df0319e1_isrForCrosstalkSources_855820_41.stderr parsl.dataflow.dflow INFO: Task 53 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec4e3f117df1a44280f910a2706c153d with result from task 53 parsl.dataflow.dflow DEBUG: Task 53 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/7e8d853c-b34c-47fb-9ef5-cb981805b8fc_isrForCrosstalkSources_855822_42.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/7e8d853c-b34c-47fb-9ef5-cb981805b8fc_isrForCrosstalkSources_855822_42.stderr parsl.dataflow.dflow INFO: Task 52 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4653d266ce2589cb29a454a11de0837e with result from task 52 parsl.dataflow.dflow DEBUG: Task 52 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/642b7cf9-266c-4336-8de4-3a63dd814f26_isrForCrosstalkSources_855911_31.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/642b7cf9-266c-4336-8de4-3a63dd814f26_isrForCrosstalkSources_855911_31.stderr parsl.dataflow.dflow INFO: Task 49 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8056ea5faf2c9a1728157671fbcaa6f5 with result from task 49 parsl.dataflow.dflow DEBUG: Task 49 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/ff960574-fef0-42a0-9498-fdb371d2639e_isrForCrosstalkSources_855911_4.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/ff960574-fef0-42a0-9498-fdb371d2639e_isrForCrosstalkSources_855911_4.stderr parsl.dataflow.dflow INFO: Task 50 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0beb5af27886ff936b4be33ca17a28c6 with result from task 50 parsl.dataflow.dflow DEBUG: Task 50 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/643159a2-4de9-46ec-b38c-dc8cd97ed166_isrForCrosstalkSources_855911_35.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/643159a2-4de9-46ec-b38c-dc8cd97ed166_isrForCrosstalkSources_855911_35.stderr parsl.dataflow.dflow INFO: Task 51 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69b8dd7af1af9467938ef540e23b9352 with result from task 51 parsl.dataflow.dflow DEBUG: Task 51 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/2ba47074-caff-40e8-8997-4cb02487999f_isrForCrosstalkSources_855820_57.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/2ba47074-caff-40e8-8997-4cb02487999f_isrForCrosstalkSources_855820_57.stderr parsl.dataflow.dflow INFO: Task 48 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b7693a2bc1ffdffe251decd97188ebf with result from task 48 parsl.dataflow.dflow DEBUG: Task 48 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/464fdfa7-9995-493e-a108-5b0561680ed9_isrForCrosstalkSources_855909_53.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/464fdfa7-9995-493e-a108-5b0561680ed9_isrForCrosstalkSources_855909_53.stderr parsl.dataflow.dflow INFO: Task 47 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b42b7ecb7227c9aac74d5dddaffa72c5 with result from task 47 parsl.dataflow.dflow DEBUG: Task 47 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/2419087c-1e24-4307-9127-639c5e0c2895_isrForCrosstalkSources_855821_54.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/2419087c-1e24-4307-9127-639c5e0c2895_isrForCrosstalkSources_855821_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:48 proc_lsst.multi:146 [INFO] found job 16968815 in provider checkpoint proc_lsst.multi INFO: found job 16968815 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:48 proc_lsst.multi:146 [INFO] found job 16968816 in provider checkpoint proc_lsst.multi INFO: found job 16968816 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:49 proc_lsst.multi:146 [INFO] found job 16968817 in provider checkpoint proc_lsst.multi INFO: found job 16968817 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.dataflow.dflow INFO: Task 60 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4daf2ce3b468d956243e38cf4c97560 with result from task 60 parsl.dataflow.dflow DEBUG: Task 60 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/ed68e1a7-2c2c-4c27-98ad-49da54e517c1_isrForCrosstalkSources_855821_25.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/ed68e1a7-2c2c-4c27-98ad-49da54e517c1_isrForCrosstalkSources_855821_25.stderr parsl.dataflow.dflow INFO: Task 59 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry caaeab4ee9da5ce348687e143f0bfdda with result from task 59 parsl.dataflow.dflow DEBUG: Task 59 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/92b66c65-3c16-423b-b29d-e40ca2a43f35_isrForCrosstalkSources_855821_51.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/92b66c65-3c16-423b-b29d-e40ca2a43f35_isrForCrosstalkSources_855821_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:49 proc_lsst.multi:146 [INFO] found job 16968818 in provider checkpoint proc_lsst.multi INFO: found job 16968818 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:50 proc_lsst.multi:146 [INFO] found job 16968819 in provider checkpoint proc_lsst.multi INFO: found job 16968819 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.dataflow.dflow INFO: Task 71 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 554ecda2dd2f8b75aaee69be124eba97 with result from task 71 parsl.dataflow.dflow DEBUG: Task 71 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/f934bc3d-0393-4300-af59-1f8428e7412f_isrForCrosstalkSources_855909_36.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/f934bc3d-0393-4300-af59-1f8428e7412f_isrForCrosstalkSources_855909_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:50 proc_lsst.multi:146 [INFO] found job 16968823 in provider checkpoint proc_lsst.multi INFO: found job 16968823 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:50 proc_lsst.multi:146 [INFO] found job 16968824 in provider checkpoint proc_lsst.multi INFO: found job 16968824 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.dataflow.dflow INFO: Task 81 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b72c01347d9d580f4f17cedc487c7e5 with result from task 81 parsl.dataflow.dflow DEBUG: Task 81 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/4d4340d5-b095-4dd9-8a7c-fef548a7fac9_isrForCrosstalkSources_855911_59.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/4d4340d5-b095-4dd9-8a7c-fef548a7fac9_isrForCrosstalkSources_855911_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:50 proc_lsst.multi:146 [INFO] found job 16968825 in provider checkpoint proc_lsst.multi INFO: found job 16968825 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:51 proc_lsst.multi:146 [INFO] found job 16968826 in provider checkpoint proc_lsst.multi INFO: found job 16968826 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:51 proc_lsst.multi:146 [INFO] found job 16968827 in provider checkpoint proc_lsst.multi INFO: found job 16968827 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:51 proc_lsst.multi:146 [INFO] found job 16968828 in provider checkpoint proc_lsst.multi INFO: found job 16968828 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:51 proc_lsst.multi:146 [INFO] found job 16968829 in provider checkpoint proc_lsst.multi INFO: found job 16968829 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:52 proc_lsst.multi:146 [INFO] found job 16968830 in provider checkpoint proc_lsst.multi INFO: found job 16968830 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:52 proc_lsst.multi:146 [INFO] found job 16968831 in provider checkpoint proc_lsst.multi INFO: found job 16968831 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:52 proc_lsst.multi:146 [INFO] found job 16968832 in provider checkpoint proc_lsst.multi INFO: found job 16968832 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.dataflow.dflow INFO: Task 72 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a850075b03dac8cbe0559b7285adca17 with result from task 72 parsl.dataflow.dflow DEBUG: Task 72 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/be0cfc9d-e3ff-4a47-af7d-ed26fe20df0b_isrForCrosstalkSources_855910_42.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/be0cfc9d-e3ff-4a47-af7d-ed26fe20df0b_isrForCrosstalkSources_855910_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:52 proc_lsst.multi:146 [INFO] found job 16968833 in provider checkpoint proc_lsst.multi INFO: found job 16968833 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.dataflow.dflow INFO: Task 179 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d64e8e2f409ab89d2ed89c0e6369c420 with result from task 179 parsl.dataflow.dflow DEBUG: Task 179 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/a70a3d07-573b-41a3-b9d9-2e0957730ec1_isrForCrosstalkSources_855820_3.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/a70a3d07-573b-41a3-b9d9-2e0957730ec1_isrForCrosstalkSources_855820_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:53 proc_lsst.multi:146 [INFO] found job 16968834 in provider checkpoint proc_lsst.multi INFO: found job 16968834 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:53 proc_lsst.multi:146 [INFO] found job 16968835 in provider checkpoint proc_lsst.multi INFO: found job 16968835 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:53 proc_lsst.multi:146 [INFO] found job 16968836 in provider checkpoint proc_lsst.multi INFO: found job 16968836 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:53 proc_lsst.multi:146 [INFO] found job 16968837 in provider checkpoint proc_lsst.multi INFO: found job 16968837 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:54 proc_lsst.multi:146 [INFO] found job 16968838 in provider checkpoint proc_lsst.multi INFO: found job 16968838 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:54 proc_lsst.multi:146 [INFO] found job 16968839 in provider checkpoint proc_lsst.multi INFO: found job 16968839 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.dataflow.dflow INFO: Task 82 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01aa5ca2157dfa8f2a4e6d5a07fac99f with result from task 82 parsl.dataflow.dflow DEBUG: Task 82 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/f1d9e07c-efbc-4dc6-aede-8300449ee33a_isrForCrosstalkSources_855820_5.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/f1d9e07c-efbc-4dc6-aede-8300449ee33a_isrForCrosstalkSources_855820_5.stderr parsl.dataflow.dflow INFO: Task 180 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a07742b5eeacb574e52064b32eb637f with result from task 180 parsl.dataflow.dflow DEBUG: Task 180 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/8344e640-81ad-47ea-b218-99018068f49b_isrForCrosstalkSources_855822_36.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/8344e640-81ad-47ea-b218-99018068f49b_isrForCrosstalkSources_855822_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:54 proc_lsst.multi:146 [INFO] found job 16968840 in provider checkpoint proc_lsst.multi INFO: found job 16968840 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:54 proc_lsst.multi:146 [INFO] found job 16968841 in provider checkpoint proc_lsst.multi INFO: found job 16968841 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:54 proc_lsst.multi:146 [INFO] found job 16968842 in provider checkpoint proc_lsst.multi INFO: found job 16968842 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:55 proc_lsst.multi:146 [INFO] found job 16968843 in provider checkpoint proc_lsst.multi INFO: found job 16968843 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.dataflow.dflow INFO: Task 181 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18f4ae4f33a677f248e3652ca0f444c8 with result from task 181 parsl.dataflow.dflow DEBUG: Task 181 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/349659b3-3964-40bd-9891-8ea26fe9f618_isrForCrosstalkSources_855820_61.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/349659b3-3964-40bd-9891-8ea26fe9f618_isrForCrosstalkSources_855820_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:55 proc_lsst.multi:146 [INFO] found job 16968844 in provider checkpoint proc_lsst.multi INFO: found job 16968844 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:55 proc_lsst.multi:146 [INFO] found job 16968845 in provider checkpoint proc_lsst.multi INFO: found job 16968845 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:55 proc_lsst.multi:146 [INFO] found job 16968846 in provider checkpoint proc_lsst.multi INFO: found job 16968846 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:55 proc_lsst.multi:146 [INFO] found job 16968847 in provider checkpoint proc_lsst.multi INFO: found job 16968847 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:56 proc_lsst.multi:146 [INFO] found job 16968848 in provider checkpoint proc_lsst.multi INFO: found job 16968848 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:56 proc_lsst.multi:146 [INFO] found job 16968849 in provider checkpoint proc_lsst.multi INFO: found job 16968849 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:56 proc_lsst.multi:146 [INFO] found job 16968850 in provider checkpoint proc_lsst.multi INFO: found job 16968850 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.dataflow.dflow INFO: Task 183 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d980291df15588c91befbd8dc114778 with result from task 183 parsl.dataflow.dflow DEBUG: Task 183 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/26e3cb7b-388f-460a-9962-06877df2a3bd_isrForCrosstalkSources_855821_47.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/26e3cb7b-388f-460a-9962-06877df2a3bd_isrForCrosstalkSources_855821_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:56 proc_lsst.multi:146 [INFO] found job 16968851 in provider checkpoint proc_lsst.multi INFO: found job 16968851 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:56 proc_lsst.multi:146 [INFO] found job 16968852 in provider checkpoint proc_lsst.multi INFO: found job 16968852 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.dataflow.dflow INFO: Task 182 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be62cc9f7e541094b9a111ea358d32f1 with result from task 182 parsl.dataflow.dflow DEBUG: Task 182 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/2553a92b-9da3-44fe-ae33-b4819965fc17_isrForCrosstalkSources_855821_39.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/2553a92b-9da3-44fe-ae33-b4819965fc17_isrForCrosstalkSources_855821_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:57 proc_lsst.multi:146 [INFO] found job 16968853 in provider checkpoint proc_lsst.multi INFO: found job 16968853 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 293 parsl.dataflow.strategy DEBUG: Executor multi has 293 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22762785682704 2024-03-09 20:50:57 proc_lsst.multi:146 [INFO] found job 35332 in provider local proc_lsst.multi INFO: found job 35332 in provider local 2024-03-09 20:50:57 proc_lsst.multi:146 [INFO] found job 16968750 in provider astro proc_lsst.multi INFO: found job 16968750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:57 proc_lsst.multi:146 [INFO] found job 16968751 in provider astro proc_lsst.multi INFO: found job 16968751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:57 proc_lsst.multi:146 [INFO] found job 16968752 in provider astro proc_lsst.multi INFO: found job 16968752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.dataflow.dflow INFO: Task 254 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9675f37b63d33f6f23d44e30772124f with result from task 254 parsl.dataflow.dflow DEBUG: Task 254 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/c825b944-98ef-46c2-999f-b7a9bae78744_isrForCrosstalkSources_855820_44.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/c825b944-98ef-46c2-999f-b7a9bae78744_isrForCrosstalkSources_855820_44.stderr parsl.dataflow.dflow INFO: Task 252 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7fdfc0085e25dfa1d701eea168d1981e with result from task 252 parsl.dataflow.dflow DEBUG: Task 252 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/f8ad3f7b-ce3d-4505-81fc-f0f76fc7a518_isrForCrosstalkSources_855820_30.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/f8ad3f7b-ce3d-4505-81fc-f0f76fc7a518_isrForCrosstalkSources_855820_30.stderr parsl.dataflow.dflow INFO: Task 253 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64b1cab9d2c693e9a4cd002fcbd3a3a1 with result from task 253 parsl.dataflow.dflow DEBUG: Task 253 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/51d755f8-588c-4187-9ad0-7be446d4c936_isrForCrosstalkSources_855820_8.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/51d755f8-588c-4187-9ad0-7be446d4c936_isrForCrosstalkSources_855820_8.stderr parsl.dataflow.dflow INFO: Task 235 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46174b62094407a78f6de0392cdcc892 with result from task 235 parsl.dataflow.dflow DEBUG: Task 235 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/b2ca9ba2-a204-42d1-9a4f-70015053a679_isrForCrosstalkSources_855911_32.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/b2ca9ba2-a204-42d1-9a4f-70015053a679_isrForCrosstalkSources_855911_32.stderr parsl.dataflow.dflow INFO: Task 233 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry baa0b2cc743ff8f551255e467a646433 with result from task 233 parsl.dataflow.dflow DEBUG: Task 233 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/780cf3cb-4093-4ebb-ae7a-dea4b7d97f25_isrForCrosstalkSources_855911_8.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/780cf3cb-4093-4ebb-ae7a-dea4b7d97f25_isrForCrosstalkSources_855911_8.stderr parsl.dataflow.dflow INFO: Task 251 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c22a2b40f728ab2d05f8b3a6f10da05c with result from task 251 parsl.dataflow.dflow DEBUG: Task 251 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/5a5bb5bd-e724-489a-b9db-3b0939427895_isrForCrosstalkSources_855910_27.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/5a5bb5bd-e724-489a-b9db-3b0939427895_isrForCrosstalkSources_855910_27.stderr parsl.dataflow.dflow INFO: Task 239 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7d855b0ff48e45c79e1ca3fd387d570 with result from task 239 parsl.dataflow.dflow DEBUG: Task 239 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/6a730bb7-2ff1-4f1a-9d90-7a369315a44b_isrForCrosstalkSources_855820_26.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/6a730bb7-2ff1-4f1a-9d90-7a369315a44b_isrForCrosstalkSources_855820_26.stderr parsl.dataflow.dflow INFO: Task 99 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 526259724f182762f55664c898432039 with result from task 99 parsl.dataflow.dflow DEBUG: Task 99 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/7b8e8dad-27f4-4a0a-98ab-e6f18f09a98f_isrForCrosstalkSources_855909_33.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/7b8e8dad-27f4-4a0a-98ab-e6f18f09a98f_isrForCrosstalkSources_855909_33.stderr parsl.dataflow.dflow INFO: Task 102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4a243ec0286a7403c61067791951dea with result from task 102 parsl.dataflow.dflow DEBUG: Task 102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/22748c6a-cc63-49a3-aaf2-eb7283d034ef_isrForCrosstalkSources_855821_34.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/22748c6a-cc63-49a3-aaf2-eb7283d034ef_isrForCrosstalkSources_855821_34.stderr parsl.dataflow.dflow INFO: Task 341 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b25cf291e0e005cb3c9a75f72c7ccb7e with result from task 341 parsl.dataflow.dflow DEBUG: Task 341 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/caaa3e28-fbd8-4413-8263-57e21b931695_isrForCrosstalkSources_855822_24.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/caaa3e28-fbd8-4413-8263-57e21b931695_isrForCrosstalkSources_855822_24.stderr parsl.dataflow.dflow INFO: Task 100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a981057fe53e9a0982c6d00ea9ed4b0 with result from task 100 parsl.dataflow.dflow DEBUG: Task 100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/194af422-dd03-447d-9358-072bb1685e12_isrForCrosstalkSources_855910_24.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/194af422-dd03-447d-9358-072bb1685e12_isrForCrosstalkSources_855910_24.stderr parsl.dataflow.dflow INFO: Task 104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab1ac6c105e10e5315a7a6aa4afcbef5 with result from task 104 parsl.dataflow.dflow DEBUG: Task 104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/6511eb27-d0ea-4b76-a030-26342eb1ae4a_isrForCrosstalkSources_855909_20.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/6511eb27-d0ea-4b76-a030-26342eb1ae4a_isrForCrosstalkSources_855909_20.stderr parsl.dataflow.dflow INFO: Task 255 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6bd04404ea4f85e9722a879c8e61a5e5 with result from task 255 parsl.dataflow.dflow DEBUG: Task 255 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/db56d21d-3cd4-4a0f-a281-252d0afbdcde_isrForCrosstalkSources_855909_14.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/db56d21d-3cd4-4a0f-a281-252d0afbdcde_isrForCrosstalkSources_855909_14.stderr parsl.dataflow.dflow INFO: Task 237 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8a8d76bb53a6ff7cf0a53438ecdc426 with result from task 237 parsl.dataflow.dflow DEBUG: Task 237 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/19891b45-e8fd-4ce9-865c-470b7ca1c5e3_isrForCrosstalkSources_855911_56.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/19891b45-e8fd-4ce9-865c-470b7ca1c5e3_isrForCrosstalkSources_855911_56.stderr parsl.dataflow.dflow INFO: Task 241 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18924f260b2a026a97db62dfa73681f7 with result from task 241 parsl.dataflow.dflow DEBUG: Task 241 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/6e67293a-0ac9-4640-b10c-b40216c2cb22_isrForCrosstalkSources_855821_19.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/6e67293a-0ac9-4640-b10c-b40216c2cb22_isrForCrosstalkSources_855821_19.stderr parsl.dataflow.dflow INFO: Task 250 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e962419f14c59f0985308b072b045095 with result from task 250 parsl.dataflow.dflow DEBUG: Task 250 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/a52868bf-f587-46cc-88c7-4c0360e106ec_isrForCrosstalkSources_855911_22.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/a52868bf-f587-46cc-88c7-4c0360e106ec_isrForCrosstalkSources_855911_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:50:59 proc_lsst.multi:146 [INFO] found job 16968753 in provider astro proc_lsst.multi INFO: found job 16968753 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.dataflow.dflow INFO: Task 227 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5cdc39a0c14300834da9896205a874c with result from task 227 parsl.dataflow.dflow DEBUG: Task 227 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/466a83ba-9ea3-4fc6-9b47-4adf479126b5_isrForCrosstalkSources_855909_22.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/466a83ba-9ea3-4fc6-9b47-4adf479126b5_isrForCrosstalkSources_855909_22.stderr parsl.dataflow.dflow INFO: Task 228 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11e5165930cbccbfe4ff94bbce9c4950 with result from task 228 parsl.dataflow.dflow DEBUG: Task 228 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/9c48a5ef-f7fd-4e6f-8267-3f8cca701161_isrForCrosstalkSources_855909_21.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/9c48a5ef-f7fd-4e6f-8267-3f8cca701161_isrForCrosstalkSources_855909_21.stderr parsl.dataflow.dflow INFO: Task 243 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96cb37ff19c1e4fae677195df23dd260 with result from task 243 parsl.dataflow.dflow DEBUG: Task 243 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/af25f7b5-27f5-4d27-bdcb-2e3c3b4e63ec_isrForCrosstalkSources_855911_21.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/af25f7b5-27f5-4d27-bdcb-2e3c3b4e63ec_isrForCrosstalkSources_855911_21.stderr parsl.dataflow.dflow INFO: Task 244 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1ba7a5cae771909a9db534e8a9576aa with result from task 244 parsl.dataflow.dflow DEBUG: Task 244 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/c902e118-4d11-4fda-b8c0-71438b9f3651_isrForCrosstalkSources_855909_11.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/c902e118-4d11-4fda-b8c0-71438b9f3651_isrForCrosstalkSources_855909_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:00 proc_lsst.multi:146 [INFO] found job 16968754 in provider astro proc_lsst.multi INFO: found job 16968754 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:00 proc_lsst.multi:146 [INFO] found job 16968755 in provider astro proc_lsst.multi INFO: found job 16968755 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:01 proc_lsst.multi:146 [INFO] found job 16968756 in provider astro proc_lsst.multi INFO: found job 16968756 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.dataflow.dflow INFO: Task 106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 302448d0ef27cefb1a0be72580bc8d26 with result from task 106 parsl.dataflow.dflow DEBUG: Task 106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/300ffd52-09fe-4c60-bc8c-ffff4986b713_isrForCrosstalkSources_855910_37.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/300ffd52-09fe-4c60-bc8c-ffff4986b713_isrForCrosstalkSources_855910_37.stderr parsl.dataflow.dflow INFO: Task 105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 663ebdd79980a6bf9f6dc31e3371f3aa with result from task 105 parsl.dataflow.dflow DEBUG: Task 105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/a27f3f12-3334-4676-8e45-4a25e4011e44_isrForCrosstalkSources_855822_2.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/a27f3f12-3334-4676-8e45-4a25e4011e44_isrForCrosstalkSources_855822_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:01 proc_lsst.multi:146 [INFO] found job 16968757 in provider astro proc_lsst.multi INFO: found job 16968757 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.dataflow.dflow INFO: Task 342 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52d83d12de658fedbfc95a5b86d8092d with result from task 342 parsl.dataflow.dflow DEBUG: Task 342 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/3fa4efd6-50b3-4e2b-bebb-8620739d25fc_isrForCrosstalkSources_855821_5.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/3fa4efd6-50b3-4e2b-bebb-8620739d25fc_isrForCrosstalkSources_855821_5.stderr parsl.dataflow.dflow INFO: Task 258 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5662247dfce05540a7cd5c9916f3345 with result from task 258 parsl.dataflow.dflow DEBUG: Task 258 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/f5ca4899-caf2-4a24-978b-90aef5dc243e_isrForCrosstalkSources_855821_29.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/f5ca4899-caf2-4a24-978b-90aef5dc243e_isrForCrosstalkSources_855821_29.stderr parsl.dataflow.dflow INFO: Task 261 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d6778eed17f290cfb093d98f617f219 with result from task 261 parsl.dataflow.dflow DEBUG: Task 261 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/9700f98f-94e9-4036-83db-69de371cebd4_isrForCrosstalkSources_855820_19.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/9700f98f-94e9-4036-83db-69de371cebd4_isrForCrosstalkSources_855820_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:01 proc_lsst.multi:146 [INFO] found job 16968758 in provider astro proc_lsst.multi INFO: found job 16968758 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.dataflow.dflow INFO: Task 245 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3ffc69088e04ec08b190a5f8d3d8b9f with result from task 245 parsl.dataflow.dflow DEBUG: Task 245 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/fedcac09-dbb8-4cbe-b21a-35673f0ac727_isrForCrosstalkSources_855820_11.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/fedcac09-dbb8-4cbe-b21a-35673f0ac727_isrForCrosstalkSources_855820_11.stderr parsl.dataflow.dflow INFO: Task 256 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b6c2fc05eb470378ed534a501a8a55c with result from task 256 parsl.dataflow.dflow DEBUG: Task 256 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/c7ca1547-1a7e-45aa-a520-8138bc98c811_isrForCrosstalkSources_855821_61.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/c7ca1547-1a7e-45aa-a520-8138bc98c811_isrForCrosstalkSources_855821_61.stderr parsl.dataflow.dflow INFO: Task 257 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fc38dcee97b961a95758c64b80dd6d6 with result from task 257 parsl.dataflow.dflow DEBUG: Task 257 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/eb391d8d-d471-4a43-88a1-51502968a348_isrForCrosstalkSources_855822_1.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/eb391d8d-d471-4a43-88a1-51502968a348_isrForCrosstalkSources_855822_1.stderr parsl.dataflow.dflow INFO: Task 238 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1457d72f465da99a7c5b409a834522a with result from task 238 parsl.dataflow.dflow DEBUG: Task 238 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/b0b34994-98d2-4065-8e9e-04e7501c51df_isrForCrosstalkSources_855822_16.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/b0b34994-98d2-4065-8e9e-04e7501c51df_isrForCrosstalkSources_855822_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:01 proc_lsst.multi:146 [INFO] found job 16968759 in provider astro proc_lsst.multi INFO: found job 16968759 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.dataflow.dflow INFO: Task 224 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1292f530adfa3ce9a8138149b791653b with result from task 224 parsl.dataflow.dflow DEBUG: Task 224 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/1ffb075b-fed2-4add-ba8c-86c287059dfc_isrForCrosstalkSources_855910_7.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/1ffb075b-fed2-4add-ba8c-86c287059dfc_isrForCrosstalkSources_855910_7.stderr parsl.dataflow.dflow INFO: Task 232 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a18891cb32a2fee5dfa37b64c356095a with result from task 232 parsl.dataflow.dflow DEBUG: Task 232 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/a1000149-39e1-4570-ab13-bd66acde7170_isrForCrosstalkSources_855910_18.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/a1000149-39e1-4570-ab13-bd66acde7170_isrForCrosstalkSources_855910_18.stderr parsl.dataflow.dflow INFO: Task 101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b1a3e85b4071666aaac42cc10b41c39 with result from task 101 parsl.dataflow.dflow DEBUG: Task 101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/0f1443a5-950b-412e-b7ad-aa490d0a0898_isrForCrosstalkSources_855909_24.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/0f1443a5-950b-412e-b7ad-aa490d0a0898_isrForCrosstalkSources_855909_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:01 proc_lsst.multi:146 [INFO] found job 16968760 in provider astro proc_lsst.multi INFO: found job 16968760 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.dataflow.dflow INFO: Task 240 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b61d8b7e1fcdec83a1c511450a85fd2 with result from task 240 parsl.dataflow.dflow DEBUG: Task 240 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/1a78c147-5622-4235-94e8-30d4cb307b8d_isrForCrosstalkSources_855820_35.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/1a78c147-5622-4235-94e8-30d4cb307b8d_isrForCrosstalkSources_855820_35.stderr parsl.dataflow.dflow INFO: Task 230 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c6c5091446c2596b11a654212609d35 with result from task 230 parsl.dataflow.dflow DEBUG: Task 230 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/ba7587a0-701d-4dda-be3d-bff46634ce11_isrForCrosstalkSources_855910_36.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/ba7587a0-701d-4dda-be3d-bff46634ce11_isrForCrosstalkSources_855910_36.stderr parsl.dataflow.dflow INFO: Task 229 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry baf8f5da895a959df4fd65088c0b4941 with result from task 229 parsl.dataflow.dflow DEBUG: Task 229 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/e83bf6f8-e63a-4308-bb6e-1cf961507c90_isrForCrosstalkSources_855821_30.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/e83bf6f8-e63a-4308-bb6e-1cf961507c90_isrForCrosstalkSources_855821_30.stderr parsl.dataflow.dflow INFO: Task 249 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e76f3648ee03743a4b214b882e8387a with result from task 249 parsl.dataflow.dflow DEBUG: Task 249 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/740059b6-a066-484f-8ac3-f5fc2a765adf_isrForCrosstalkSources_855910_3.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/740059b6-a066-484f-8ac3-f5fc2a765adf_isrForCrosstalkSources_855910_3.stderr parsl.dataflow.dflow INFO: Task 242 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 635927ca59f516e6620d3114c28920f8 with result from task 242 parsl.dataflow.dflow DEBUG: Task 242 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/671a4b04-d910-4966-b110-43a9709669fe_isrForCrosstalkSources_855821_41.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/671a4b04-d910-4966-b110-43a9709669fe_isrForCrosstalkSources_855821_41.stderr parsl.dataflow.dflow INFO: Task 246 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 633dcc9e25f85728bc6d71694172d24e with result from task 246 parsl.dataflow.dflow DEBUG: Task 246 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/d42b9f39-6104-4398-85f1-4de0b57ad068_isrForCrosstalkSources_855821_28.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/d42b9f39-6104-4398-85f1-4de0b57ad068_isrForCrosstalkSources_855821_28.stderr parsl.dataflow.dflow INFO: Task 103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae8ce2a1a77e418000fb07f7e7cadc6c with result from task 103 parsl.dataflow.dflow DEBUG: Task 103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/47a06708-b3b2-4f48-bacc-cc36bb60a7c0_isrForCrosstalkSources_855821_58.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/47a06708-b3b2-4f48-bacc-cc36bb60a7c0_isrForCrosstalkSources_855821_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:02 proc_lsst.multi:146 [INFO] found job 16968761 in provider astro proc_lsst.multi INFO: found job 16968761 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:02 proc_lsst.multi:146 [INFO] found job 16968762 in provider astro proc_lsst.multi INFO: found job 16968762 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:02 proc_lsst.multi:146 [INFO] found job 16968763 in provider astro proc_lsst.multi INFO: found job 16968763 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.dataflow.dflow INFO: Task 268 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry acd8ea4cfd0f2bb4aeedc02dd4a04b4a with result from task 268 parsl.dataflow.dflow DEBUG: Task 268 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/cddd1985-a124-4ef2-a88f-a93a93da4608_isrForCrosstalkSources_855910_56.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/cddd1985-a124-4ef2-a88f-a93a93da4608_isrForCrosstalkSources_855910_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:02 proc_lsst.multi:146 [INFO] found job 16968764 in provider astro proc_lsst.multi INFO: found job 16968764 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.dataflow.dflow INFO: Task 248 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35ea7d9a4c5484abac34aeb9afeca9ce with result from task 248 parsl.dataflow.dflow DEBUG: Task 248 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/e12df680-210c-42d6-8e38-f2d5640e4683_isrForCrosstalkSources_855821_37.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/e12df680-210c-42d6-8e38-f2d5640e4683_isrForCrosstalkSources_855821_37.stderr parsl.dataflow.dflow INFO: Task 247 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 409be12ae56016383709e63626f14b3d with result from task 247 parsl.dataflow.dflow DEBUG: Task 247 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/bd6e23c6-f3f8-4b53-9c9e-4242db2a6a99_isrForCrosstalkSources_855910_57.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/bd6e23c6-f3f8-4b53-9c9e-4242db2a6a99_isrForCrosstalkSources_855910_57.stderr parsl.dataflow.dflow INFO: Task 234 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a30b2184473938f85c7279d05d88a0d with result from task 234 parsl.dataflow.dflow DEBUG: Task 234 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/6eb70db8-fa64-42d3-88bb-32dea8d37670_isrForCrosstalkSources_855822_47.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/6eb70db8-fa64-42d3-88bb-32dea8d37670_isrForCrosstalkSources_855822_47.stderr parsl.dataflow.dflow INFO: Task 231 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2f14720c5c3f2b09cbac03cb535e45b with result from task 231 parsl.dataflow.dflow DEBUG: Task 231 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/9c0168d8-aba6-4cfe-bdfd-8c19785da501_isrForCrosstalkSources_855821_55.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/9c0168d8-aba6-4cfe-bdfd-8c19785da501_isrForCrosstalkSources_855821_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Task 225 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.memoization DEBUG: Storing app cache entry 188653b6103c19db4a483444273cc661 with result from task 225 parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow DEBUG: Task 225 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:02 proc_lsst.multi:146 [INFO] found job 16968765 in provider astro proc_lsst.multi INFO: found job 16968765 in provider astro parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/ed477725-8c22-424d-9f6c-8b377a2e3a53_isrForCrosstalkSources_855910_52.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/ed477725-8c22-424d-9f6c-8b377a2e3a53_isrForCrosstalkSources_855910_52.stderr parsl.dataflow.dflow INFO: Task 226 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5905dddd1e5051311ca2056aed03a82a with result from task 226 parsl.dataflow.dflow DEBUG: Task 226 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/84cc07c0-550b-472c-8306-6f140be05b77_isrForCrosstalkSources_855821_44.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/84cc07c0-550b-472c-8306-6f140be05b77_isrForCrosstalkSources_855821_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:03 proc_lsst.multi:146 [INFO] found job 16968766 in provider astro proc_lsst.multi INFO: found job 16968766 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.dataflow.dflow INFO: Task 260 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d39753135d8bc3aec65b89dca7a1be65 with result from task 260 parsl.dataflow.dflow DEBUG: Task 260 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/da5faeda-d22f-4b6c-83fa-413a5272ea12_isrForCrosstalkSources_855822_11.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/da5faeda-d22f-4b6c-83fa-413a5272ea12_isrForCrosstalkSources_855822_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:03 proc_lsst.multi:146 [INFO] found job 16968767 in provider astro proc_lsst.multi INFO: found job 16968767 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.dataflow.dflow INFO: Task 262 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ecf54c57e80c4f47f27905e5c0b7a5fb with result from task 262 parsl.dataflow.dflow DEBUG: Task 262 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/52fcfbf8-9559-4cf4-b6d2-b561115bedf8_isrForCrosstalkSources_855822_43.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/52fcfbf8-9559-4cf4-b6d2-b561115bedf8_isrForCrosstalkSources_855822_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:03 proc_lsst.multi:146 [INFO] found job 16968768 in provider astro proc_lsst.multi INFO: found job 16968768 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.dataflow.dflow INFO: Task 263 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 282021e2dd2b25f708dc9950f1761c18 with result from task 263 parsl.dataflow.dflow DEBUG: Task 263 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/1486ec87-2b6f-4cfe-b27b-ebafb46995bb_isrForCrosstalkSources_855909_3.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/1486ec87-2b6f-4cfe-b27b-ebafb46995bb_isrForCrosstalkSources_855909_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:03 proc_lsst.multi:146 [INFO] found job 16968769 in provider astro proc_lsst.multi INFO: found job 16968769 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:03 proc_lsst.multi:146 [INFO] found job 16968770 in provider astro proc_lsst.multi INFO: found job 16968770 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.dataflow.dflow INFO: Task 236 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c48ed28e36aae8f9afa0ad87a137b644 with result from task 236 parsl.dataflow.dflow DEBUG: Task 236 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/55f226b8-143e-491f-9576-bcf776b95423_isrForCrosstalkSources_855909_48.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/55f226b8-143e-491f-9576-bcf776b95423_isrForCrosstalkSources_855909_48.stderr parsl.dataflow.dflow INFO: Task 265 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a337ffb91de0888a20ec67858b751b96 with result from task 265 parsl.dataflow.dflow DEBUG: Task 265 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/c8629cb5-601d-44ed-987c-4c42d22fee26_isrForCrosstalkSources_855909_31.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/c8629cb5-601d-44ed-987c-4c42d22fee26_isrForCrosstalkSources_855909_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Task 266 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.memoization DEBUG: Storing app cache entry 2d6dd667ca13c2e78ed6f9e094c10d28 with result from task 266 parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow DEBUG: Task 266 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/8ec91fd1-fdc6-4c00-be1a-20bde715d59c_isrForCrosstalkSources_855821_36.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/8ec91fd1-fdc6-4c00-be1a-20bde715d59c_isrForCrosstalkSources_855821_36.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:04 proc_lsst.multi:146 [INFO] found job 16968771 in provider astro proc_lsst.multi INFO: found job 16968771 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.dataflow.dflow INFO: Task 259 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 832fe61760a3d275415b16271f58e6e7 with result from task 259 parsl.dataflow.dflow DEBUG: Task 259 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/79e36655-1696-40ca-8161-193745171f30_isrForCrosstalkSources_855821_57.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/79e36655-1696-40ca-8161-193745171f30_isrForCrosstalkSources_855821_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:04 proc_lsst.multi:146 [INFO] found job 16968772 in provider astro proc_lsst.multi INFO: found job 16968772 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.dataflow.dflow INFO: Task 267 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8541532df6063a31a0045e241a6fa24e with result from task 267 parsl.dataflow.dflow DEBUG: Task 267 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/57a0d132-9bf2-429a-8ce1-f43040a878e5_isrForCrosstalkSources_855909_55.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/57a0d132-9bf2-429a-8ce1-f43040a878e5_isrForCrosstalkSources_855909_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:04 proc_lsst.multi:146 [INFO] found job 16968773 in provider astro proc_lsst.multi INFO: found job 16968773 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.dataflow.dflow INFO: Task 264 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55eb33ffa641d7f8ea4e7ea972d4b72f with result from task 264 parsl.dataflow.dflow DEBUG: Task 264 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/6cb51672-26d7-452f-bb94-2880e713542a_isrForCrosstalkSources_855911_54.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/6cb51672-26d7-452f-bb94-2880e713542a_isrForCrosstalkSources_855911_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:04 proc_lsst.multi:146 [INFO] found job 16968774 in provider astro proc_lsst.multi INFO: found job 16968774 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:04 proc_lsst.multi:146 [INFO] found job 16968775 in provider astro proc_lsst.multi INFO: found job 16968775 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:05 proc_lsst.multi:146 [INFO] found job 16968776 in provider astro proc_lsst.multi INFO: found job 16968776 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:05 proc_lsst.multi:146 [INFO] found job 16968777 in provider astro proc_lsst.multi INFO: found job 16968777 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:05 proc_lsst.multi:146 [INFO] found job 16968778 in provider astro proc_lsst.multi INFO: found job 16968778 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:05 proc_lsst.multi:146 [INFO] found job 16968779 in provider astro proc_lsst.multi INFO: found job 16968779 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:05 proc_lsst.multi:146 [INFO] found job 16968780 in provider escience_rtx6k proc_lsst.multi INFO: found job 16968780 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968780,16968781,16968782,16968783,16968784' parsl.providers.slurm.slurm DEBUG: squeue returned 16968781 R 16968782 R 16968783 R 16968784 R 16968780 R parsl.providers.slurm.slurm DEBUG: Updating job 16968781 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968780 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:06 proc_lsst.multi:146 [INFO] found job 16968781 in provider escience_rtx6k proc_lsst.multi INFO: found job 16968781 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968780,16968781,16968782,16968783,16968784' parsl.providers.slurm.slurm DEBUG: squeue returned 16968781 R 16968782 R 16968783 R 16968784 R 16968780 R parsl.providers.slurm.slurm DEBUG: Updating job 16968781 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968780 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:06 proc_lsst.multi:146 [INFO] found job 16968782 in provider escience_rtx6k proc_lsst.multi INFO: found job 16968782 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968780,16968781,16968782,16968783,16968784' parsl.providers.slurm.slurm DEBUG: squeue returned 16968781 R 16968782 R 16968783 R 16968784 R 16968780 R parsl.providers.slurm.slurm DEBUG: Updating job 16968781 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968780 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:06 proc_lsst.multi:146 [INFO] found job 16968783 in provider escience_rtx6k proc_lsst.multi INFO: found job 16968783 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968780,16968781,16968782,16968783,16968784' parsl.dataflow.dflow INFO: Task 85 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc439cf4fe347d9c51f6b302d1c45516 with result from task 85 parsl.dataflow.dflow DEBUG: Task 85 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/ca81d67f-d5a1-4267-9c29-358164dcbe46_isrForCrosstalkSources_855820_39.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/ca81d67f-d5a1-4267-9c29-358164dcbe46_isrForCrosstalkSources_855820_39.stderr parsl.dataflow.dflow INFO: Task 88 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e7af0eb634cd1c919cb61d3d00f7410 with result from task 88 parsl.dataflow.dflow DEBUG: Task 88 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/fe488e0c-208f-425b-9f7e-6f89b078cea6_isrForCrosstalkSources_855822_26.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/fe488e0c-208f-425b-9f7e-6f89b078cea6_isrForCrosstalkSources_855822_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968781 R 16968782 R 16968783 R 16968784 R 16968780 R parsl.providers.slurm.slurm DEBUG: Updating job 16968781 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968780 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:06 proc_lsst.multi:146 [INFO] found job 16968784 in provider escience_rtx6k proc_lsst.multi INFO: found job 16968784 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968780,16968781,16968782,16968783,16968784' parsl.dataflow.dflow INFO: Task 78 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44d2b33123904841f7db9b2f5132aafc with result from task 78 parsl.dataflow.dflow DEBUG: Task 78 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/94d982f3-bf5c-4d8d-836a-2224bb0ccd0a_isrForCrosstalkSources_855911_27.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/94d982f3-bf5c-4d8d-836a-2224bb0ccd0a_isrForCrosstalkSources_855911_27.stderr parsl.dataflow.dflow INFO: Task 77 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de10a393bd01fc73c454b4090287caab with result from task 77 parsl.dataflow.dflow DEBUG: Task 77 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/e7c44aa8-34c2-4441-b956-1659a280d74f_isrForCrosstalkSources_855820_20.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/e7c44aa8-34c2-4441-b956-1659a280d74f_isrForCrosstalkSources_855820_20.stderr parsl.dataflow.dflow INFO: Task 75 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f99d286b182fa4b033f40b25b73385ca with result from task 75 parsl.dataflow.dflow DEBUG: Task 75 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/18c5d542-aed9-4caa-8381-f31e3d9b6c27_isrForCrosstalkSources_855820_12.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/18c5d542-aed9-4caa-8381-f31e3d9b6c27_isrForCrosstalkSources_855820_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968781 R 16968782 R 16968783 R 16968784 R 16968780 R parsl.providers.slurm.slurm DEBUG: Updating job 16968781 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968780 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:06 proc_lsst.multi:146 [INFO] found job 16968785 in provider escience_a40 proc_lsst.multi INFO: found job 16968785 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:07 proc_lsst.multi:146 [INFO] found job 16968786 in provider escience_a40 proc_lsst.multi INFO: found job 16968786 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:07 proc_lsst.multi:146 [INFO] found job 16968787 in provider escience_a40 proc_lsst.multi INFO: found job 16968787 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.dataflow.dflow INFO: Task 80 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b5966306eeac768a816ad14ab6bf8de with result from task 80 parsl.dataflow.dflow DEBUG: Task 80 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/bcb4dda4-98d2-44a5-a91e-ce820b3ae440_isrForCrosstalkSources_855910_44.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/bcb4dda4-98d2-44a5-a91e-ce820b3ae440_isrForCrosstalkSources_855910_44.stderr parsl.dataflow.dflow INFO: Task 79 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6309f15117fe9cdd3e18de2a8fc17e4 with result from task 79 parsl.dataflow.dflow DEBUG: Task 79 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/4469e5fa-e768-4c49-ab9c-a305448d873d_isrForCrosstalkSources_855910_17.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/4469e5fa-e768-4c49-ab9c-a305448d873d_isrForCrosstalkSources_855910_17.stderr parsl.dataflow.dflow INFO: Task 90 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ade40d9875fe99a883fd0832e624150f with result from task 90 parsl.dataflow.dflow DEBUG: Task 90 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/5708f642-fbdb-4f57-9dd3-62a76bf926f7_isrForCrosstalkSources_855910_43.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/5708f642-fbdb-4f57-9dd3-62a76bf926f7_isrForCrosstalkSources_855910_43.stderr parsl.dataflow.dflow INFO: Task 84 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c565e8428dd0ca19a7c6d0c48c9391e7 with result from task 84 parsl.dataflow.dflow DEBUG: Task 84 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/160ba715-7ba2-40e0-acdd-35c81c3b5aaa_isrForCrosstalkSources_855910_22.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/160ba715-7ba2-40e0-acdd-35c81c3b5aaa_isrForCrosstalkSources_855910_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:07 proc_lsst.multi:146 [INFO] found job 16968788 in provider escience_a40 proc_lsst.multi INFO: found job 16968788 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.dataflow.dflow INFO: Task 83 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aaeaafa53e9b80d1dcddfe7864cb74a0 with result from task 83 parsl.dataflow.dflow DEBUG: Task 83 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/e94695f0-0e84-4fed-82d0-ab7d9ea10207_isrForCrosstalkSources_855822_18.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/e94695f0-0e84-4fed-82d0-ab7d9ea10207_isrForCrosstalkSources_855822_18.stderr parsl.dataflow.dflow INFO: Task 73 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c62cc86e46d4efc7e8810f526925eda3 with result from task 73 parsl.dataflow.dflow DEBUG: Task 73 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/46d28b07-69fd-4e9a-aac8-ee83845f8493_isrForCrosstalkSources_855822_30.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/46d28b07-69fd-4e9a-aac8-ee83845f8493_isrForCrosstalkSources_855822_30.stderr parsl.dataflow.dflow INFO: Task 87 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62f16fb7e20934a1dc2abdda6d01c5bf with result from task 87 parsl.dataflow.dflow DEBUG: Task 87 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/b1105cc8-1b72-41ea-8fe2-09c6b3a0f242_isrForCrosstalkSources_855822_5.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/b1105cc8-1b72-41ea-8fe2-09c6b3a0f242_isrForCrosstalkSources_855822_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:07 proc_lsst.multi:146 [INFO] found job 16968789 in provider escience_a40 proc_lsst.multi INFO: found job 16968789 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.dataflow.dflow INFO: Task 89 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33011f3b432635dca6450d43112961c5 with result from task 89 parsl.dataflow.dflow DEBUG: Task 89 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/a5180d8f-c3f0-4b68-828b-4c6214ff9657_isrForCrosstalkSources_855911_6.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/a5180d8f-c3f0-4b68-828b-4c6214ff9657_isrForCrosstalkSources_855911_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:07 proc_lsst.multi:146 [INFO] found job 16968790 in provider escience_a40 proc_lsst.multi INFO: found job 16968790 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.dataflow.dflow INFO: Task 74 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5aa7884cc0bd7a7a636b41789810f1b3 with result from task 74 parsl.dataflow.dflow DEBUG: Task 74 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/8b9850eb-befa-45b0-9184-b985ac830bd4_isrForCrosstalkSources_855820_38.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/8b9850eb-befa-45b0-9184-b985ac830bd4_isrForCrosstalkSources_855820_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:08 proc_lsst.multi:146 [INFO] found job 16968791 in provider escience_a40 proc_lsst.multi INFO: found job 16968791 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:08 proc_lsst.multi:146 [INFO] found job 16968792 in provider escience_a40 proc_lsst.multi INFO: found job 16968792 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:08 proc_lsst.multi:146 [INFO] found job 16968793 in provider escience_a40 proc_lsst.multi INFO: found job 16968793 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.dataflow.dflow INFO: Task 76 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7769508b06b563f13032287adab4c110 with result from task 76 parsl.dataflow.dflow DEBUG: Task 76 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/4c3bd2cd-9569-4805-a375-b70d04370148_isrForCrosstalkSources_855909_30.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/4c3bd2cd-9569-4805-a375-b70d04370148_isrForCrosstalkSources_855909_30.stderr parsl.dataflow.dflow INFO: Task 86 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a47780112dd9fab791edaf37b3bbc08 with result from task 86 parsl.dataflow.dflow DEBUG: Task 86 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/d3ad82ff-4bd6-43fa-9f6c-af4a8c9681b4_isrForCrosstalkSources_855909_60.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/d3ad82ff-4bd6-43fa-9f6c-af4a8c9681b4_isrForCrosstalkSources_855909_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:08 proc_lsst.multi:146 [INFO] found job 16968794 in provider escience_a40 proc_lsst.multi INFO: found job 16968794 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:08 proc_lsst.multi:146 [INFO] found job 16968795 in provider escience_a40 proc_lsst.multi INFO: found job 16968795 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:09 proc_lsst.multi:146 [INFO] found job 16968796 in provider escience_a40 proc_lsst.multi INFO: found job 16968796 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:09 proc_lsst.multi:146 [INFO] found job 16968797 in provider escience_a40 proc_lsst.multi INFO: found job 16968797 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:09 proc_lsst.multi:146 [INFO] found job 16968798 in provider escience_a40 proc_lsst.multi INFO: found job 16968798 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:09 proc_lsst.multi:146 [INFO] found job 16968799 in provider escience_a40 proc_lsst.multi INFO: found job 16968799 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:09 proc_lsst.multi:146 [INFO] found job 16968800 in provider escience_a40 proc_lsst.multi INFO: found job 16968800 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:10 proc_lsst.multi:146 [INFO] found job 16968805 in provider escience_a40 proc_lsst.multi INFO: found job 16968805 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:10 proc_lsst.multi:146 [INFO] found job 16968811 in provider escience_a40 proc_lsst.multi INFO: found job 16968811 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:10 proc_lsst.multi:146 [INFO] found job 16968812 in provider escience_a40 proc_lsst.multi INFO: found job 16968812 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:10 proc_lsst.multi:146 [INFO] found job 16968813 in provider escience_a40 proc_lsst.multi INFO: found job 16968813 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:10 proc_lsst.multi:146 [INFO] found job 16968814 in provider checkpoint proc_lsst.multi INFO: found job 16968814 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:11 proc_lsst.multi:146 [INFO] found job 16968815 in provider checkpoint proc_lsst.multi INFO: found job 16968815 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:11 proc_lsst.multi:146 [INFO] found job 16968816 in provider checkpoint proc_lsst.multi INFO: found job 16968816 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:11 proc_lsst.multi:146 [INFO] found job 16968817 in provider checkpoint proc_lsst.multi INFO: found job 16968817 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.dataflow.dflow INFO: Task 91 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7bd7873d6c38c31e85428772afd5400b with result from task 91 parsl.dataflow.dflow DEBUG: Task 91 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/6c6e9c99-b821-43c5-9db7-b70e7cf8db82_isrForCrosstalkSources_855822_14.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/6c6e9c99-b821-43c5-9db7-b70e7cf8db82_isrForCrosstalkSources_855822_14.stderr parsl.dataflow.dflow INFO: Task 96 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c017cb5e7db1781fed7f6eac2ea2a909 with result from task 96 parsl.dataflow.dflow DEBUG: Task 96 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/0af4fd2f-6336-421c-8660-ad747b62b8f6_isrForCrosstalkSources_855822_56.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/0af4fd2f-6336-421c-8660-ad747b62b8f6_isrForCrosstalkSources_855822_56.stderr parsl.dataflow.dflow INFO: Task 95 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e9d773488bdbb080b8df98123fab84a with result from task 95 parsl.dataflow.dflow DEBUG: Task 95 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/d740cb42-bf17-4d95-8836-c147ab3ac4f7_isrForCrosstalkSources_855911_2.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/d740cb42-bf17-4d95-8836-c147ab3ac4f7_isrForCrosstalkSources_855911_2.stderr parsl.dataflow.dflow INFO: Task 98 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7f3df18c4fb4dd3c4437fe672bcaab7 with result from task 98 parsl.dataflow.dflow DEBUG: Task 98 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/e69b0fde-51a2-4394-bd40-c6eb2747fd0e_isrForCrosstalkSources_855820_17.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/e69b0fde-51a2-4394-bd40-c6eb2747fd0e_isrForCrosstalkSources_855820_17.stderr parsl.dataflow.dflow INFO: Task 92 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9556a094525006bd09a58bd44e6c1b55 with result from task 92 parsl.dataflow.dflow DEBUG: Task 92 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/1edddfda-8049-4327-992d-7e5543e27889_isrForCrosstalkSources_855910_10.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/1edddfda-8049-4327-992d-7e5543e27889_isrForCrosstalkSources_855910_10.stderr parsl.dataflow.dflow INFO: Task 93 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94fb5bfe4034cc96acbe1b82c5bfa31c with result from task 93 parsl.dataflow.dflow DEBUG: Task 93 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/e5b231ed-9851-41a7-8c3f-0a57f24d1c5a_isrForCrosstalkSources_855822_6.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/e5b231ed-9851-41a7-8c3f-0a57f24d1c5a_isrForCrosstalkSources_855822_6.stderr parsl.dataflow.dflow INFO: Task 97 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ea5342be757156fb198f513e61f131e with result from task 97 parsl.dataflow.dflow DEBUG: Task 97 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/caa0eb0a-0c43-43b7-8c3d-302f9cc823f2_isrForCrosstalkSources_855910_32.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/caa0eb0a-0c43-43b7-8c3d-302f9cc823f2_isrForCrosstalkSources_855910_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:11 proc_lsst.multi:146 [INFO] found job 16968818 in provider checkpoint proc_lsst.multi INFO: found job 16968818 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:11 proc_lsst.multi:146 [INFO] found job 16968819 in provider checkpoint proc_lsst.multi INFO: found job 16968819 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.dataflow.dflow INFO: Task 94 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c0b213cbbeb73deb9528374f1b038dd with result from task 94 parsl.dataflow.dflow DEBUG: Task 94 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/bd8cc624-8612-4c13-b0e3-706c88b6310b_isrForCrosstalkSources_855822_38.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/bd8cc624-8612-4c13-b0e3-706c88b6310b_isrForCrosstalkSources_855822_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:12 proc_lsst.multi:146 [INFO] found job 16968823 in provider checkpoint proc_lsst.multi INFO: found job 16968823 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:12 proc_lsst.multi:146 [INFO] found job 16968824 in provider checkpoint proc_lsst.multi INFO: found job 16968824 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:12 proc_lsst.multi:146 [INFO] found job 16968825 in provider checkpoint proc_lsst.multi INFO: found job 16968825 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:12 proc_lsst.multi:146 [INFO] found job 16968826 in provider checkpoint proc_lsst.multi INFO: found job 16968826 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:12 proc_lsst.multi:146 [INFO] found job 16968827 in provider checkpoint proc_lsst.multi INFO: found job 16968827 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:13 proc_lsst.multi:146 [INFO] found job 16968828 in provider checkpoint proc_lsst.multi INFO: found job 16968828 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:13 proc_lsst.multi:146 [INFO] found job 16968829 in provider checkpoint proc_lsst.multi INFO: found job 16968829 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:13 proc_lsst.multi:146 [INFO] found job 16968830 in provider checkpoint proc_lsst.multi INFO: found job 16968830 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.dataflow.dflow INFO: Task 116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d914e1ade9feb882344b6dc4de47f9ff with result from task 116 parsl.dataflow.dflow DEBUG: Task 116 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/b95aa468-c53e-4189-b775-a235a57215e0_isrForCrosstalkSources_855911_7.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/b95aa468-c53e-4189-b775-a235a57215e0_isrForCrosstalkSources_855911_7.stderr parsl.dataflow.dflow INFO: Task 122 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0bfe2a89c8c6c87a90e41ff8bfe972aa with result from task 122 parsl.dataflow.dflow DEBUG: Task 122 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/983b57d9-d701-4447-919b-fab33cf21818_isrForCrosstalkSources_855821_14.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/983b57d9-d701-4447-919b-fab33cf21818_isrForCrosstalkSources_855821_14.stderr parsl.dataflow.dflow INFO: Task 115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 139a94c01137cfe28829aacc6aae4dab with result from task 115 parsl.dataflow.dflow DEBUG: Task 115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/76d9ff32-c0c7-44fb-b014-58f9f4546a5e_isrForCrosstalkSources_855821_12.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/76d9ff32-c0c7-44fb-b014-58f9f4546a5e_isrForCrosstalkSources_855821_12.stderr parsl.dataflow.dflow INFO: Task 117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3f2f8cc86a254dfb9517c9ccd68b80c with result from task 117 parsl.dataflow.dflow DEBUG: Task 117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/09851928-2ab7-4cbd-b189-5ae91e72dc79_isrForCrosstalkSources_855821_6.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/09851928-2ab7-4cbd-b189-5ae91e72dc79_isrForCrosstalkSources_855821_6.stderr parsl.dataflow.dflow INFO: Task 121 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d63c8526d60321457c4aa1ed750e900 with result from task 121 parsl.dataflow.dflow DEBUG: Task 121 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/3d1340c7-7322-45d9-950b-f731f19ef39a_isrForCrosstalkSources_855910_14.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/3d1340c7-7322-45d9-950b-f731f19ef39a_isrForCrosstalkSources_855910_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:13 proc_lsst.multi:146 [INFO] found job 16968831 in provider checkpoint proc_lsst.multi INFO: found job 16968831 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.dataflow.dflow INFO: Task 120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58f9e74a98d625bc50710534dfb8d697 with result from task 120 parsl.dataflow.dflow DEBUG: Task 120 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/6465ec59-9a5f-44f2-b67c-a1e63452b2c2_isrForCrosstalkSources_855910_47.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/6465ec59-9a5f-44f2-b67c-a1e63452b2c2_isrForCrosstalkSources_855910_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:13 proc_lsst.multi:146 [INFO] found job 16968832 in provider checkpoint proc_lsst.multi INFO: found job 16968832 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:14 proc_lsst.multi:146 [INFO] found job 16968833 in provider checkpoint proc_lsst.multi INFO: found job 16968833 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:14 proc_lsst.multi:146 [INFO] found job 16968834 in provider checkpoint proc_lsst.multi INFO: found job 16968834 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.dataflow.dflow INFO: Task 119 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8828440444a6692a2737c7f9369e81f with result from task 119 parsl.dataflow.dflow DEBUG: Task 119 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/ad9ca9de-09a1-4a10-b92e-c75ce37471bf_isrForCrosstalkSources_855909_52.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/ad9ca9de-09a1-4a10-b92e-c75ce37471bf_isrForCrosstalkSources_855909_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:14 proc_lsst.multi:146 [INFO] found job 16968835 in provider checkpoint proc_lsst.multi INFO: found job 16968835 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:14 proc_lsst.multi:146 [INFO] found job 16968836 in provider checkpoint proc_lsst.multi INFO: found job 16968836 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.dataflow.dflow INFO: Task 118 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da1d3246864f79e679fb633991de51fe with result from task 118 parsl.dataflow.dflow DEBUG: Task 118 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/30ebd418-98b8-4074-b9be-b22fca161ebb_isrForCrosstalkSources_855909_43.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/30ebd418-98b8-4074-b9be-b22fca161ebb_isrForCrosstalkSources_855909_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:15 proc_lsst.multi:146 [INFO] found job 16968837 in provider checkpoint proc_lsst.multi INFO: found job 16968837 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:15 proc_lsst.multi:146 [INFO] found job 16968838 in provider checkpoint proc_lsst.multi INFO: found job 16968838 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:15 proc_lsst.multi:146 [INFO] found job 16968839 in provider checkpoint proc_lsst.multi INFO: found job 16968839 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:15 proc_lsst.multi:146 [INFO] found job 16968840 in provider checkpoint proc_lsst.multi INFO: found job 16968840 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.dataflow.dflow INFO: Task 166 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04f83bbcab22385dcaa07d021425faf3 with result from task 166 parsl.dataflow.dflow DEBUG: Task 166 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/4980235f-c3b9-4dd4-a46d-3a2207661509_isrForCrosstalkSources_855909_26.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/4980235f-c3b9-4dd4-a46d-3a2207661509_isrForCrosstalkSources_855909_26.stderr parsl.dataflow.dflow INFO: Task 174 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ad6ecb273afd22792c209ddf7437de5 with result from task 174 parsl.dataflow.dflow DEBUG: Task 174 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/14329ff8-9520-4ac2-9e79-b14b2862d5c3_isrForCrosstalkSources_855821_27.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/14329ff8-9520-4ac2-9e79-b14b2862d5c3_isrForCrosstalkSources_855821_27.stderr parsl.dataflow.dflow INFO: Task 172 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca4b338c55dca5747e3fa69d0ea95c20 with result from task 172 parsl.dataflow.dflow DEBUG: Task 172 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/3ada3f31-68ef-4ed8-9f82-53d373c2b504_isrForCrosstalkSources_855910_33.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/3ada3f31-68ef-4ed8-9f82-53d373c2b504_isrForCrosstalkSources_855910_33.stderr parsl.dataflow.dflow INFO: Task 175 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 941441e10bd0737336fc4596ca88443d with result from task 175 parsl.dataflow.dflow DEBUG: Task 175 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/9366051c-cb15-4d4c-9d6f-05b84dae8156_isrForCrosstalkSources_855820_27.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/9366051c-cb15-4d4c-9d6f-05b84dae8156_isrForCrosstalkSources_855820_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:15 proc_lsst.multi:146 [INFO] found job 16968841 in provider checkpoint proc_lsst.multi INFO: found job 16968841 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:15 proc_lsst.multi:146 [INFO] found job 16968842 in provider checkpoint proc_lsst.multi INFO: found job 16968842 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:16 proc_lsst.multi:146 [INFO] found job 16968843 in provider checkpoint proc_lsst.multi INFO: found job 16968843 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.dataflow.dflow INFO: Task 178 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09c9daf3694aa212f35dd552558372cd with result from task 178 parsl.dataflow.dflow DEBUG: Task 178 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/12191924-91e5-473f-b886-4eabc8e03a06_isrForCrosstalkSources_855821_4.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/12191924-91e5-473f-b886-4eabc8e03a06_isrForCrosstalkSources_855821_4.stderr parsl.dataflow.dflow INFO: Task 170 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9fcd39cd035845ad9f614a7a09f32d9d with result from task 170 parsl.dataflow.dflow DEBUG: Task 170 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/7148c205-e875-404c-a644-661b6acda94f_isrForCrosstalkSources_855821_17.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/7148c205-e875-404c-a644-661b6acda94f_isrForCrosstalkSources_855821_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:16 proc_lsst.multi:146 [INFO] found job 16968844 in provider checkpoint proc_lsst.multi INFO: found job 16968844 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:16 proc_lsst.multi:146 [INFO] found job 16968845 in provider checkpoint proc_lsst.multi INFO: found job 16968845 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.dataflow.dflow INFO: Task 176 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03bb741e7cd973ac7e6abdc90de42da3 with result from task 176 parsl.dataflow.dflow DEBUG: Task 176 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/56187f5b-d739-4f75-87a4-645cbc9a1827_isrForCrosstalkSources_855909_41.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/56187f5b-d739-4f75-87a4-645cbc9a1827_isrForCrosstalkSources_855909_41.stderr parsl.dataflow.dflow INFO: Task 169 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c249ca2b518a76967dfe694a31a68177 with result from task 169 parsl.dataflow.dflow DEBUG: Task 169 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/8ae7e14a-6c23-413a-93f7-8b185bee19d6_isrForCrosstalkSources_855909_18.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/8ae7e14a-6c23-413a-93f7-8b185bee19d6_isrForCrosstalkSources_855909_18.stderr parsl.dataflow.dflow INFO: Task 168 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51be36d3c7057a8f9426e1fe7d8a2e54 with result from task 168 parsl.dataflow.dflow DEBUG: Task 168 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/d91531a8-c97c-4ed6-8765-fa35798ee4db_isrForCrosstalkSources_855911_24.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/d91531a8-c97c-4ed6-8765-fa35798ee4db_isrForCrosstalkSources_855911_24.stderr parsl.dataflow.dflow INFO: Task 163 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a5825609c3c2d4827ad69aaec431e9b with result from task 163 parsl.dataflow.dflow DEBUG: Task 163 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/1723b53a-e927-40ff-9c0b-77ba8c663bac_isrForCrosstalkSources_855821_3.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/1723b53a-e927-40ff-9c0b-77ba8c663bac_isrForCrosstalkSources_855821_3.stderr parsl.dataflow.dflow INFO: Task 177 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 491b867208d284c56c71ebed2c59a8c6 with result from task 177 parsl.dataflow.dflow DEBUG: Task 177 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/7d178d57-5c50-4060-b06b-2c606e49e0cd_isrForCrosstalkSources_855909_61.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/7d178d57-5c50-4060-b06b-2c606e49e0cd_isrForCrosstalkSources_855909_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:16 proc_lsst.multi:146 [INFO] found job 16968846 in provider checkpoint proc_lsst.multi INFO: found job 16968846 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.dataflow.dflow INFO: Task 167 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fb04d1e92ad5e6e083a0c89f0ed911b with result from task 167 parsl.dataflow.dflow DEBUG: Task 167 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/c8a77bb7-5b75-40d5-a0c3-3c1b5f659ce4_isrForCrosstalkSources_855821_2.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/c8a77bb7-5b75-40d5-a0c3-3c1b5f659ce4_isrForCrosstalkSources_855821_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:17 proc_lsst.multi:146 [INFO] found job 16968847 in provider checkpoint proc_lsst.multi INFO: found job 16968847 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:17 proc_lsst.multi:146 [INFO] found job 16968848 in provider checkpoint proc_lsst.multi INFO: found job 16968848 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.dataflow.dflow INFO: Task 173 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3afaf031ec53a4fb7107d49e3d45b36e with result from task 173 parsl.dataflow.dflow DEBUG: Task 173 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/182a5abe-5aaf-4534-9354-2468bfc97a30_isrForCrosstalkSources_855910_6.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/182a5abe-5aaf-4534-9354-2468bfc97a30_isrForCrosstalkSources_855910_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:17 proc_lsst.multi:146 [INFO] found job 16968849 in provider checkpoint proc_lsst.multi INFO: found job 16968849 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.dataflow.dflow INFO: Task 165 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34f2b16acf0c6055b6f6f60875f08844 with result from task 165 parsl.dataflow.dflow DEBUG: Task 165 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/91c9eab0-d671-4271-86bd-64743e9b600b_isrForCrosstalkSources_855820_22.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/91c9eab0-d671-4271-86bd-64743e9b600b_isrForCrosstalkSources_855820_22.stderr parsl.dataflow.dflow INFO: Task 164 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f376f7965280759bec1610e25b7847f with result from task 164 parsl.dataflow.dflow DEBUG: Task 164 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/e576b03a-2a40-4b2b-a242-6434db2a7219_isrForCrosstalkSources_855822_4.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/e576b03a-2a40-4b2b-a242-6434db2a7219_isrForCrosstalkSources_855822_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:17 proc_lsst.multi:146 [INFO] found job 16968850 in provider checkpoint proc_lsst.multi INFO: found job 16968850 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.dataflow.dflow INFO: Task 159 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fcfbe7ebd37346e6c10bad29640ef4f0 with result from task 159 parsl.dataflow.dflow DEBUG: Task 159 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/60f679ef-3b94-4266-b907-149f5c577053_isrForCrosstalkSources_855911_34.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/60f679ef-3b94-4266-b907-149f5c577053_isrForCrosstalkSources_855911_34.stderr parsl.dataflow.dflow INFO: Task 162 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1de4cb593fb57d7fcddfc51582357d43 with result from task 162 parsl.dataflow.dflow DEBUG: Task 162 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/46751165-d85d-48c6-bcff-fd91a79a0d61_isrForCrosstalkSources_855911_1.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/46751165-d85d-48c6-bcff-fd91a79a0d61_isrForCrosstalkSources_855911_1.stderr parsl.dataflow.dflow INFO: Task 156 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2aa4c1b434eb62c105ece206e0dfeaf4 with result from task 156 parsl.dataflow.dflow DEBUG: Task 156 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/a147c4a9-9115-4a68-8536-68f1b82c7011_isrForCrosstalkSources_855822_45.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/a147c4a9-9115-4a68-8536-68f1b82c7011_isrForCrosstalkSources_855822_45.stderr parsl.dataflow.dflow INFO: Task 161 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66053d82bd47cb65a6d475d2a85941ff with result from task 161 parsl.dataflow.dflow DEBUG: Task 161 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/c41cfc8e-fc60-48c6-a8c2-f31e1760dd40_isrForCrosstalkSources_855821_53.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/c41cfc8e-fc60-48c6-a8c2-f31e1760dd40_isrForCrosstalkSources_855821_53.stderr parsl.dataflow.dflow INFO: Task 157 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1742e6ca80ff07050ab9a70eae84cd13 with result from task 157 parsl.dataflow.dflow DEBUG: Task 157 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/c92885f0-3f33-49e2-8de5-e7c643078c3e_isrForCrosstalkSources_855821_18.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/c92885f0-3f33-49e2-8de5-e7c643078c3e_isrForCrosstalkSources_855821_18.stderr parsl.dataflow.dflow INFO: Task 160 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2449d97553feaadce03bb20627cf67c2 with result from task 160 parsl.dataflow.dflow DEBUG: Task 160 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/f3089494-b88b-45e6-9045-651586aa7c32_isrForCrosstalkSources_855911_30.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/f3089494-b88b-45e6-9045-651586aa7c32_isrForCrosstalkSources_855911_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:17 proc_lsst.multi:146 [INFO] found job 16968851 in provider checkpoint proc_lsst.multi INFO: found job 16968851 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.dataflow.dflow INFO: Task 171 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ae91de8f379bb756d482c35f9e47d7f with result from task 171 parsl.dataflow.dflow DEBUG: Task 171 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/081d6598-d26e-47f7-a585-19d5bf276c17_isrForCrosstalkSources_855820_37.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/081d6598-d26e-47f7-a585-19d5bf276c17_isrForCrosstalkSources_855820_37.stderr parsl.dataflow.dflow INFO: Task 158 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd7f9c7776857a571d19a1f250850fd0 with result from task 158 parsl.dataflow.dflow DEBUG: Task 158 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/5f695f4b-33ba-40ad-87e4-30cf51cc961b_isrForCrosstalkSources_855909_28.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/5f695f4b-33ba-40ad-87e4-30cf51cc961b_isrForCrosstalkSources_855909_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:18 proc_lsst.multi:146 [INFO] found job 16968852 in provider checkpoint proc_lsst.multi INFO: found job 16968852 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:18 proc_lsst.multi:146 [INFO] found job 16968853 in provider checkpoint proc_lsst.multi INFO: found job 16968853 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.dataflow.dflow INFO: Task 126 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef7ae0b9601d4f706557385acff0921c with result from task 126 parsl.dataflow.dflow DEBUG: Task 126 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/2b85aa75-02eb-4f68-84f2-24c9f73d0745_isrForCrosstalkSources_855822_13.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/2b85aa75-02eb-4f68-84f2-24c9f73d0745_isrForCrosstalkSources_855822_13.stderr parsl.dataflow.dflow INFO: Task 123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52c4003cb47012f7a089d81c1f201575 with result from task 123 parsl.dataflow.dflow DEBUG: Task 123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/447c03b9-153d-4248-a444-ef511e9ca82e_isrForCrosstalkSources_855911_47.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/447c03b9-153d-4248-a444-ef511e9ca82e_isrForCrosstalkSources_855911_47.stderr parsl.dataflow.dflow INFO: Task 129 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ceb43a42e17eedb65129e4369966b3a with result from task 129 parsl.dataflow.dflow DEBUG: Task 129 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/e3bf6294-edc7-4775-afc6-c70037a29eb8_isrForCrosstalkSources_855822_32.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/e3bf6294-edc7-4775-afc6-c70037a29eb8_isrForCrosstalkSources_855822_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 180 parsl.dataflow.strategy DEBUG: Executor multi has 180 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22762785682704 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 180 parsl.dataflow.strategy DEBUG: Executor multi has 180 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22762785682704 parsl.dataflow.dflow INFO: Task 130 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e272a2a9f5e6558feab8c7c8212e583e with result from task 130 parsl.dataflow.dflow DEBUG: Task 130 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/0381c584-cae5-45e7-97fc-d9840ba3736b_isrForCrosstalkSources_855911_5.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/0381c584-cae5-45e7-97fc-d9840ba3736b_isrForCrosstalkSources_855911_5.stderr parsl.dataflow.dflow INFO: Task 127 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b89340b2ced1921aa4570c85b4da9642 with result from task 127 parsl.dataflow.dflow DEBUG: Task 127 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/fc788203-a9fb-4812-9fe3-7b1487be49c6_isrForCrosstalkSources_855909_17.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/fc788203-a9fb-4812-9fe3-7b1487be49c6_isrForCrosstalkSources_855909_17.stderr parsl.dataflow.dflow INFO: Task 155 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e2982b176ffa42950b41c2c1e5e8643 with result from task 155 parsl.dataflow.dflow DEBUG: Task 155 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/0e1ab3f7-ebe2-46d5-8169-4fc9d83e5ccf_isrForCrosstalkSources_855910_53.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/0e1ab3f7-ebe2-46d5-8169-4fc9d83e5ccf_isrForCrosstalkSources_855910_53.stderr parsl.dataflow.dflow INFO: Task 125 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 897c44d30c101ee91726ba047be3c583 with result from task 125 parsl.dataflow.dflow DEBUG: Task 125 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/039bdf69-bf88-4239-904e-fbe538b340c6_isrForCrosstalkSources_855909_47.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/039bdf69-bf88-4239-904e-fbe538b340c6_isrForCrosstalkSources_855909_47.stderr parsl.dataflow.dflow INFO: Task 128 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18f68e35ebcc1b5329107f2bd4b1a58b with result from task 128 parsl.dataflow.dflow DEBUG: Task 128 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/d675f7d8-4aeb-4a98-a053-fbee038635a1_isrForCrosstalkSources_855820_52.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/d675f7d8-4aeb-4a98-a053-fbee038635a1_isrForCrosstalkSources_855820_52.stderr parsl.dataflow.dflow INFO: Task 189 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59a6b0686e764a1937e866011ad011f8 with result from task 189 parsl.dataflow.dflow DEBUG: Task 189 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/d8bb0496-f0e9-43b4-a194-5c7e5d222660_isrForCrosstalkSources_855911_18.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/d8bb0496-f0e9-43b4-a194-5c7e5d222660_isrForCrosstalkSources_855911_18.stderr parsl.dataflow.dflow INFO: Task 188 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02b5619dfe77e079ff11c5c83f0a3967 with result from task 188 parsl.dataflow.dflow DEBUG: Task 188 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/1ed07551-3258-44aa-aa51-2cc1ab0146dd_isrForCrosstalkSources_855910_51.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/1ed07551-3258-44aa-aa51-2cc1ab0146dd_isrForCrosstalkSources_855910_51.stderr parsl.dataflow.dflow INFO: Task 124 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 752be8fefadf89977c9cbf693256430c with result from task 124 parsl.dataflow.dflow DEBUG: Task 124 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/71040246-eed5-48c9-85a9-4a59d7c0c429_isrForCrosstalkSources_855821_31.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/71040246-eed5-48c9-85a9-4a59d7c0c429_isrForCrosstalkSources_855821_31.stderr parsl.dataflow.dflow INFO: Task 186 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4dfa77ce34de3d2cf77e217a7baf1826 with result from task 186 parsl.dataflow.dflow DEBUG: Task 186 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/2a9b5761-40a2-42ca-b299-83b98e3aedd8_isrForCrosstalkSources_855821_23.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/2a9b5761-40a2-42ca-b299-83b98e3aedd8_isrForCrosstalkSources_855821_23.stderr parsl.dataflow.dflow INFO: Task 190 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 646ac596b6ad1f99fdcad5fde3ba20ab with result from task 190 parsl.dataflow.dflow DEBUG: Task 190 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/87fd7787-c531-4204-ae51-131762f53bf5_isrForCrosstalkSources_855911_36.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/87fd7787-c531-4204-ae51-131762f53bf5_isrForCrosstalkSources_855911_36.stderr parsl.dataflow.dflow INFO: Task 187 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04edbe5cb8ab6b0223981f2fd1ba08b4 with result from task 187 parsl.dataflow.dflow DEBUG: Task 187 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/056c8cf9-32f8-465e-9e1b-80d5287e8943_isrForCrosstalkSources_855821_21.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/056c8cf9-32f8-465e-9e1b-80d5287e8943_isrForCrosstalkSources_855821_21.stderr parsl.dataflow.dflow INFO: Task 132 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4fb851258d4f5df5ccbcbd3b0b4690d3 with result from task 132 parsl.dataflow.dflow DEBUG: Task 132 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/e0cad1e0-5f58-42ed-81f9-abc4c3671b1f_isrForCrosstalkSources_855822_37.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/e0cad1e0-5f58-42ed-81f9-abc4c3671b1f_isrForCrosstalkSources_855822_37.stderr parsl.dataflow.dflow INFO: Task 135 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98ae6369c2be5628eaa23bc8d7a0b9b9 with result from task 135 parsl.dataflow.dflow DEBUG: Task 135 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/9a19a9b5-58f1-4dca-9195-3485182ee451_isrForCrosstalkSources_855911_37.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/9a19a9b5-58f1-4dca-9195-3485182ee451_isrForCrosstalkSources_855911_37.stderr parsl.dataflow.dflow INFO: Task 138 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry edb1af494666620000aaa94c0c14f592 with result from task 138 parsl.dataflow.dflow DEBUG: Task 138 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/fe27fb3e-eea6-49ac-8c39-d2f16feee535_isrForCrosstalkSources_855820_24.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/fe27fb3e-eea6-49ac-8c39-d2f16feee535_isrForCrosstalkSources_855820_24.stderr parsl.dataflow.dflow INFO: Task 133 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de24fa06bfdd383ad9ec8ce2bd96e8ec with result from task 133 parsl.dataflow.dflow DEBUG: Task 133 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/3baf1cf1-455a-46c3-a9d6-bd6eee64cc98_isrForCrosstalkSources_855820_31.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/3baf1cf1-455a-46c3-a9d6-bd6eee64cc98_isrForCrosstalkSources_855820_31.stderr parsl.dataflow.dflow INFO: Task 137 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a300248e1b10291fa00bf50ae4a19f86 with result from task 137 parsl.dataflow.dflow DEBUG: Task 137 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/5cdb4000-8389-429a-b6f2-0f89617695e9_isrForCrosstalkSources_855822_17.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/5cdb4000-8389-429a-b6f2-0f89617695e9_isrForCrosstalkSources_855822_17.stderr parsl.dataflow.dflow INFO: Task 131 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b966f570d0eac8c422f043bd5abf351e with result from task 131 parsl.dataflow.dflow DEBUG: Task 131 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/845dc36b-07b1-43e1-90d9-928a88029388_isrForCrosstalkSources_855821_9.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/845dc36b-07b1-43e1-90d9-928a88029388_isrForCrosstalkSources_855821_9.stderr parsl.dataflow.dflow INFO: Task 136 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a687a01bc8a8959d14fcc8d4eda7a76e with result from task 136 parsl.dataflow.dflow DEBUG: Task 136 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/51e03e44-e3cf-42b5-8e75-d661d19c250e_isrForCrosstalkSources_855910_11.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/51e03e44-e3cf-42b5-8e75-d661d19c250e_isrForCrosstalkSources_855910_11.stderr parsl.dataflow.dflow INFO: Task 144 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92e408376dc8e304de86ac7607839bfe with result from task 144 parsl.dataflow.dflow DEBUG: Task 144 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/8c85f8ac-c36d-48d2-8b06-2ce0e0065448_isrForCrosstalkSources_855909_9.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/8c85f8ac-c36d-48d2-8b06-2ce0e0065448_isrForCrosstalkSources_855909_9.stderr parsl.dataflow.dflow INFO: Task 146 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 836ef952510d4dffee6e8bcc03a3a8ee with result from task 146 parsl.dataflow.dflow DEBUG: Task 146 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/e6aaddc9-fd8f-49be-84dc-20c5c4629ce1_isrForCrosstalkSources_855911_38.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/e6aaddc9-fd8f-49be-84dc-20c5c4629ce1_isrForCrosstalkSources_855911_38.stderr parsl.dataflow.dflow INFO: Task 147 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a91aa42740b5c162c9dd708b24f51c1d with result from task 147 parsl.dataflow.dflow DEBUG: Task 147 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/db3a81ac-34e1-4aad-9cbc-996485944580_isrForCrosstalkSources_855909_51.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/db3a81ac-34e1-4aad-9cbc-996485944580_isrForCrosstalkSources_855909_51.stderr parsl.dataflow.dflow INFO: Task 153 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52cd70407d70a39820fe523e94dffb19 with result from task 153 parsl.dataflow.dflow DEBUG: Task 153 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/d430a535-e9aa-4ef0-8017-6da8669b3bac_isrForCrosstalkSources_855909_5.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/d430a535-e9aa-4ef0-8017-6da8669b3bac_isrForCrosstalkSources_855909_5.stderr parsl.dataflow.dflow INFO: Task 149 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb020c78ed846b3baa3ee63f75e3f867 with result from task 149 parsl.dataflow.dflow DEBUG: Task 149 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/6cf38398-1445-45f5-89c8-da4cf16d7374_isrForCrosstalkSources_855821_16.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/6cf38398-1445-45f5-89c8-da4cf16d7374_isrForCrosstalkSources_855821_16.stderr parsl.dataflow.dflow INFO: Task 152 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae402526a23e0bd44a6b6f9147a3d6b9 with result from task 152 parsl.dataflow.dflow DEBUG: Task 152 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/534fb3f3-53a7-44a0-8cfd-a989ad29de4c_isrForCrosstalkSources_855910_21.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/534fb3f3-53a7-44a0-8cfd-a989ad29de4c_isrForCrosstalkSources_855910_21.stderr parsl.dataflow.dflow INFO: Task 151 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c677d87582828103716217667e97c9b1 with result from task 151 parsl.dataflow.dflow DEBUG: Task 151 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/dfca7045-e482-4bd2-8c18-5858d7329c8a_isrForCrosstalkSources_855820_4.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/dfca7045-e482-4bd2-8c18-5858d7329c8a_isrForCrosstalkSources_855820_4.stderr parsl.dataflow.dflow INFO: Task 140 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2cbb3e4a4fe1a690fac57ec5f2a69313 with result from task 140 parsl.dataflow.dflow DEBUG: Task 140 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/30ac3309-3ae6-412b-b4cf-d2deb8f7c973_isrForCrosstalkSources_855911_48.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/30ac3309-3ae6-412b-b4cf-d2deb8f7c973_isrForCrosstalkSources_855911_48.stderr parsl.dataflow.dflow INFO: Task 142 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6186b9751b47e9cf4c32d780d409668 with result from task 142 parsl.dataflow.dflow DEBUG: Task 142 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/a4998646-e1f2-44da-a68b-94c33a400be4_isrForCrosstalkSources_855822_20.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/a4998646-e1f2-44da-a68b-94c33a400be4_isrForCrosstalkSources_855822_20.stderr parsl.dataflow.dflow INFO: Task 145 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 383deadd7a707c5cc7fc6a1e0a1ccf2a with result from task 145 parsl.dataflow.dflow DEBUG: Task 145 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/5b6b3b77-2cb3-46a4-9f1b-19057856adc9_isrForCrosstalkSources_855911_50.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/5b6b3b77-2cb3-46a4-9f1b-19057856adc9_isrForCrosstalkSources_855911_50.stderr parsl.dataflow.dflow INFO: Task 139 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37ac4135c3b4efd1a11a137ee7e40c05 with result from task 139 parsl.dataflow.dflow DEBUG: Task 139 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/6b45ddb3-dec1-410a-8fb9-f79a986ea269_isrForCrosstalkSources_855909_4.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/6b45ddb3-dec1-410a-8fb9-f79a986ea269_isrForCrosstalkSources_855909_4.stderr parsl.dataflow.dflow INFO: Task 134 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f7486bdb803c7c2294b6976c6e25d84 with result from task 134 parsl.dataflow.dflow DEBUG: Task 134 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/d316f3cc-7f2a-45ad-bf4a-695401912528_isrForCrosstalkSources_855910_48.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/d316f3cc-7f2a-45ad-bf4a-695401912528_isrForCrosstalkSources_855910_48.stderr parsl.dataflow.dflow INFO: Task 112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53a718b6d56d56139786b490d8643dc0 with result from task 112 parsl.dataflow.dflow DEBUG: Task 112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/3563190b-efa4-4097-80a2-0a88ed68ed85_isrForCrosstalkSources_855910_39.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/3563190b-efa4-4097-80a2-0a88ed68ed85_isrForCrosstalkSources_855910_39.stderr parsl.dataflow.dflow INFO: Task 113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cdeb689f5dfb653e3b7e7b3e401ea78b with result from task 113 parsl.dataflow.dflow DEBUG: Task 113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/cbb64a1e-5a75-4779-a08d-87075a7a0651_isrForCrosstalkSources_855820_29.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/cbb64a1e-5a75-4779-a08d-87075a7a0651_isrForCrosstalkSources_855820_29.stderr parsl.dataflow.dflow INFO: Task 110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 062fde19b3aa7a4225bddcc2cc36b75a with result from task 110 parsl.dataflow.dflow DEBUG: Task 110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/fcc3a775-e860-4609-9a27-506efd9919b1_isrForCrosstalkSources_855910_28.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/fcc3a775-e860-4609-9a27-506efd9919b1_isrForCrosstalkSources_855910_28.stderr parsl.dataflow.dflow INFO: Task 107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32bd7f106f861555b90917b168b3da0b with result from task 107 parsl.dataflow.dflow DEBUG: Task 107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/dccb49ac-546a-4d82-a10c-60ed178995d8_isrForCrosstalkSources_855909_15.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/dccb49ac-546a-4d82-a10c-60ed178995d8_isrForCrosstalkSources_855909_15.stderr parsl.dataflow.dflow INFO: Task 114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d717c0ee1b45c19ce038dc00f6ffa12 with result from task 114 parsl.dataflow.dflow DEBUG: Task 114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/513f68d2-6d7d-43e3-92be-f4c56d2ffb21_isrForCrosstalkSources_855909_34.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/513f68d2-6d7d-43e3-92be-f4c56d2ffb21_isrForCrosstalkSources_855909_34.stderr parsl.dataflow.dflow INFO: Task 108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 407f6c8f49ba3d2628419dd6062a7ff3 with result from task 108 parsl.dataflow.dflow DEBUG: Task 108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/5740cecc-a40c-42b3-8d8b-951f6c97bc76_isrForCrosstalkSources_855821_56.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/5740cecc-a40c-42b3-8d8b-951f6c97bc76_isrForCrosstalkSources_855821_56.stderr parsl.dataflow.dflow INFO: Task 109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8d9ccd3a8deb03a2188b3fe6e4ac067 with result from task 109 parsl.dataflow.dflow DEBUG: Task 109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/d2f7d4bf-ae71-4be3-bf7c-854e7f9332e4_isrForCrosstalkSources_855911_9.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/d2f7d4bf-ae71-4be3-bf7c-854e7f9332e4_isrForCrosstalkSources_855911_9.stderr parsl.dataflow.dflow INFO: Task 111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e09ae6dccb1b0f70da982773addcaeb8 with result from task 111 parsl.dataflow.dflow DEBUG: Task 111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/89b3074c-8e17-44e6-b968-829afac8baeb_isrForCrosstalkSources_855910_49.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/89b3074c-8e17-44e6-b968-829afac8baeb_isrForCrosstalkSources_855910_49.stderr parsl.dataflow.dflow INFO: Task 184 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f05423ad6766143356a50969a56140d with result from task 184 parsl.dataflow.dflow DEBUG: Task 184 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/233d4275-25ae-47d9-8c00-07493a2e3548_isrForCrosstalkSources_855821_48.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/233d4275-25ae-47d9-8c00-07493a2e3548_isrForCrosstalkSources_855821_48.stderr parsl.dataflow.dflow INFO: Task 143 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a4cdc36f7a362ac35ec545727b979dd with result from task 143 parsl.dataflow.dflow DEBUG: Task 143 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/ceb1f1e8-be09-422f-b984-66ad8211013e_isrForCrosstalkSources_855821_62.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/ceb1f1e8-be09-422f-b984-66ad8211013e_isrForCrosstalkSources_855821_62.stderr parsl.dataflow.dflow INFO: Task 150 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3cfc768f6d121d1fbab87e55a3929a8c with result from task 150 parsl.dataflow.dflow DEBUG: Task 150 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/d8159da8-653f-4552-8923-08359445607b_isrForCrosstalkSources_855821_52.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/d8159da8-653f-4552-8923-08359445607b_isrForCrosstalkSources_855821_52.stderr parsl.dataflow.dflow INFO: Task 191 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 809efbad1121da079deff44f03e2d4ad with result from task 191 parsl.dataflow.dflow DEBUG: Task 191 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/5601c7a6-39e4-462f-8ac0-95ebb3809aaf_isrForCrosstalkSources_855820_36.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/5601c7a6-39e4-462f-8ac0-95ebb3809aaf_isrForCrosstalkSources_855820_36.stderr parsl.dataflow.dflow INFO: Task 154 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52550898a900e6c91f21123670378b7a with result from task 154 parsl.dataflow.dflow DEBUG: Task 154 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/8b79b70a-2fb3-414f-aefd-b2b5f6cfb8dd_isrForCrosstalkSources_855910_38.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/8b79b70a-2fb3-414f-aefd-b2b5f6cfb8dd_isrForCrosstalkSources_855910_38.stderr parsl.dataflow.dflow INFO: Task 141 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 895f8ef3a45db32c387b72b4ed257713 with result from task 141 parsl.dataflow.dflow DEBUG: Task 141 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/6248229d-b28b-458b-8f8f-f46cc42b9667_isrForCrosstalkSources_855909_44.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/6248229d-b28b-458b-8f8f-f46cc42b9667_isrForCrosstalkSources_855909_44.stderr parsl.dataflow.dflow INFO: Task 148 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bdea59d309b1c284b9f851e7aaec2149 with result from task 148 parsl.dataflow.dflow DEBUG: Task 148 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/c9f2dae1-405e-4f2a-96eb-55d192d731a5_isrForCrosstalkSources_855822_29.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/c9f2dae1-405e-4f2a-96eb-55d192d731a5_isrForCrosstalkSources_855822_29.stderr parsl.dataflow.dflow INFO: Task 185 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44bf0f021a7ef8d79891ee81d4c1aad4 with result from task 185 parsl.dataflow.dflow DEBUG: Task 185 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/6d58b8c2-cdd5-4242-abd7-9a16527f0a00_isrForCrosstalkSources_855822_62.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/6d58b8c2-cdd5-4242-abd7-9a16527f0a00_isrForCrosstalkSources_855822_62.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 134 parsl.dataflow.strategy DEBUG: Executor multi has 134 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22762785682704 parsl.dataflow.dflow INFO: Task 354 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96d44f02d893c625d8e0254b605a3824 with result from task 354 parsl.dataflow.dflow DEBUG: Task 354 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/77aff1c3-68a2-46a7-9208-3f44cd80223a_isrForCrosstalkSources_855822_55.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/77aff1c3-68a2-46a7-9208-3f44cd80223a_isrForCrosstalkSources_855822_55.stderr parsl.dataflow.dflow INFO: Task 351 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8474f73a01d981622b4b2e2602ca4527 with result from task 351 parsl.dataflow.dflow DEBUG: Task 351 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/4fa3ae39-b7e9-4a0f-9b3f-607ef49e6efc_isrForCrosstalkSources_855910_34.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/4fa3ae39-b7e9-4a0f-9b3f-607ef49e6efc_isrForCrosstalkSources_855910_34.stderr parsl.dataflow.dflow INFO: Task 358 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07af143984c6989eb453de9471bbd443 with result from task 358 parsl.dataflow.dflow DEBUG: Task 358 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/4c20dbe4-8153-4636-a972-f6ca52a94ddc_isrForCrosstalkSources_855821_45.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/4c20dbe4-8153-4636-a972-f6ca52a94ddc_isrForCrosstalkSources_855821_45.stderr parsl.dataflow.dflow INFO: Task 356 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d63bd59e7dfb019cd25d2fcd884408bb with result from task 356 parsl.dataflow.dflow DEBUG: Task 356 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/d9cab460-65d9-491e-be97-f15c91123674_isrForCrosstalkSources_855911_29.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/d9cab460-65d9-491e-be97-f15c91123674_isrForCrosstalkSources_855911_29.stderr parsl.dataflow.dflow INFO: Task 352 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f1076e35be5d0e785765c0cb1fc159e with result from task 352 parsl.dataflow.dflow DEBUG: Task 352 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/42d37109-06a6-4c1e-9e26-77ebd8f94e8b_isrForCrosstalkSources_855822_19.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/42d37109-06a6-4c1e-9e26-77ebd8f94e8b_isrForCrosstalkSources_855822_19.stderr parsl.dataflow.dflow INFO: Task 357 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65fac8d524909fb15519bfe4c5fa9a24 with result from task 357 parsl.dataflow.dflow DEBUG: Task 357 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/b70aefe4-ffbc-492f-92e7-11b92054c5f7_isrForCrosstalkSources_855822_7.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/b70aefe4-ffbc-492f-92e7-11b92054c5f7_isrForCrosstalkSources_855822_7.stderr parsl.dataflow.dflow INFO: Task 355 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d64bb15f9208fb42b43dfec3d5c409f with result from task 355 parsl.dataflow.dflow DEBUG: Task 355 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/ba6e7f2a-e001-46f4-a68e-7225d388c7c7_isrForCrosstalkSources_855821_43.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/ba6e7f2a-e001-46f4-a68e-7225d388c7c7_isrForCrosstalkSources_855821_43.stderr parsl.dataflow.dflow INFO: Task 353 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9dfc7d1b555a461b38739e7259bf3d96 with result from task 353 parsl.dataflow.dflow DEBUG: Task 353 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/6bf1df54-5d7f-476f-a759-9fb6f9e083fb_isrForCrosstalkSources_855910_58.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/6bf1df54-5d7f-476f-a759-9fb6f9e083fb_isrForCrosstalkSources_855910_58.stderr 2024-03-09 20:51:28 proc_lsst.multi:146 [INFO] found job 35332 in provider local proc_lsst.multi INFO: found job 35332 in provider local 2024-03-09 20:51:28 proc_lsst.multi:146 [INFO] found job 16968750 in provider astro proc_lsst.multi INFO: found job 16968750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:28 proc_lsst.multi:146 [INFO] found job 16968751 in provider astro proc_lsst.multi INFO: found job 16968751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:28 proc_lsst.multi:146 [INFO] found job 16968752 in provider astro proc_lsst.multi INFO: found job 16968752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:29 proc_lsst.multi:146 [INFO] found job 16968753 in provider astro proc_lsst.multi INFO: found job 16968753 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:29 proc_lsst.multi:146 [INFO] found job 16968754 in provider astro proc_lsst.multi INFO: found job 16968754 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:29 proc_lsst.multi:146 [INFO] found job 16968755 in provider astro proc_lsst.multi INFO: found job 16968755 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:29 proc_lsst.multi:146 [INFO] found job 16968756 in provider astro proc_lsst.multi INFO: found job 16968756 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:29 proc_lsst.multi:146 [INFO] found job 16968757 in provider astro proc_lsst.multi INFO: found job 16968757 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:30 proc_lsst.multi:146 [INFO] found job 16968758 in provider astro proc_lsst.multi INFO: found job 16968758 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:30 proc_lsst.multi:146 [INFO] found job 16968759 in provider astro proc_lsst.multi INFO: found job 16968759 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:30 proc_lsst.multi:146 [INFO] found job 16968760 in provider astro proc_lsst.multi INFO: found job 16968760 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:30 proc_lsst.multi:146 [INFO] found job 16968761 in provider astro proc_lsst.multi INFO: found job 16968761 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.dataflow.dflow INFO: Task 223 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85d823e9eac509934403fa2cf73cc89d with result from task 223 parsl.dataflow.dflow DEBUG: Task 223 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/68a2873b-0e09-4bfb-819b-59cd145804f1_isrForCrosstalkSources_855910_55.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/68a2873b-0e09-4bfb-819b-59cd145804f1_isrForCrosstalkSources_855910_55.stderr parsl.dataflow.dflow INFO: Task 214 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b8c147d7441f34f025bb282435ee0d4 with result from task 214 parsl.dataflow.dflow DEBUG: Task 214 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/3a39aaaa-54cb-4195-a393-fcd691fe6fe7_isrForCrosstalkSources_855910_12.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/3a39aaaa-54cb-4195-a393-fcd691fe6fe7_isrForCrosstalkSources_855910_12.stderr parsl.dataflow.dflow INFO: Task 211 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ae855a17ce2bbae663e98124efd25e0 with result from task 211 parsl.dataflow.dflow DEBUG: Task 211 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/a302223e-e22e-4972-9905-ac51244fd8dc_isrForCrosstalkSources_855910_4.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/a302223e-e22e-4972-9905-ac51244fd8dc_isrForCrosstalkSources_855910_4.stderr parsl.dataflow.dflow INFO: Task 213 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18c2312cba1096adfb88a30db8eaa1c1 with result from task 213 parsl.dataflow.dflow DEBUG: Task 213 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/0110a7fb-4d00-4a53-87b7-5de31374ca04_isrForCrosstalkSources_855909_46.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/0110a7fb-4d00-4a53-87b7-5de31374ca04_isrForCrosstalkSources_855909_46.stderr parsl.dataflow.dflow INFO: Task 209 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6859c838ae3fa0619817ed1f2a8d336 with result from task 209 parsl.dataflow.dflow DEBUG: Task 209 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/a98c121a-be59-4a36-8ecd-8abff1739aae_isrForCrosstalkSources_855910_20.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/a98c121a-be59-4a36-8ecd-8abff1739aae_isrForCrosstalkSources_855910_20.stderr parsl.dataflow.dflow INFO: Task 208 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15cb9428402118666d4e4c1ff606d7b1 with result from task 208 parsl.dataflow.dflow DEBUG: Task 208 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/df91e2f6-fca8-47a6-8a45-faef366e2720_isrForCrosstalkSources_855820_46.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/df91e2f6-fca8-47a6-8a45-faef366e2720_isrForCrosstalkSources_855820_46.stderr parsl.dataflow.dflow INFO: Task 215 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e4b80fffc9910fa102e4f3ccfb9199e with result from task 215 parsl.dataflow.dflow DEBUG: Task 215 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/c087b1d8-58d0-489a-97ee-0f11048667a0_isrForCrosstalkSources_855820_50.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/c087b1d8-58d0-489a-97ee-0f11048667a0_isrForCrosstalkSources_855820_50.stderr parsl.dataflow.dflow INFO: Task 212 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a57db97d0fb7ed3ad9586e36dfa42194 with result from task 212 parsl.dataflow.dflow DEBUG: Task 212 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/6098c0e0-4e62-4e68-82fe-ae76d834013f_isrForCrosstalkSources_855909_1.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/6098c0e0-4e62-4e68-82fe-ae76d834013f_isrForCrosstalkSources_855909_1.stderr parsl.dataflow.dflow INFO: Task 210 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry adbcd830dcc2efaa31223beabab3002f with result from task 210 parsl.dataflow.dflow DEBUG: Task 210 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/dd8c6e99-39ba-4c67-9932-19951b10011e_isrForCrosstalkSources_855822_8.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/dd8c6e99-39ba-4c67-9932-19951b10011e_isrForCrosstalkSources_855822_8.stderr parsl.dataflow.dflow INFO: Task 218 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c16f7d2a9f99a49e128af80d47336773 with result from task 218 parsl.dataflow.dflow DEBUG: Task 218 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/9f7fde05-8983-45f2-9379-7436cf8d050a_isrForCrosstalkSources_855909_19.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/9f7fde05-8983-45f2-9379-7436cf8d050a_isrForCrosstalkSources_855909_19.stderr parsl.dataflow.dflow INFO: Task 222 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d4a69e71af489c165f166c081279112 with result from task 222 parsl.dataflow.dflow DEBUG: Task 222 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/ece1d310-d293-4b16-bbb4-324cb01623f3_isrForCrosstalkSources_855909_12.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/ece1d310-d293-4b16-bbb4-324cb01623f3_isrForCrosstalkSources_855909_12.stderr parsl.dataflow.dflow INFO: Task 216 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62578a5c2ce295ad79b8838665b24e64 with result from task 216 parsl.dataflow.dflow DEBUG: Task 216 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/0e7b35a7-ede7-49aa-b802-ff407484ff4b_isrForCrosstalkSources_855820_2.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/0e7b35a7-ede7-49aa-b802-ff407484ff4b_isrForCrosstalkSources_855820_2.stderr parsl.dataflow.dflow INFO: Task 219 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82e7263418100a2c829c344ab69872be with result from task 219 parsl.dataflow.dflow DEBUG: Task 219 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/acc25d0b-d59b-44b2-9bd3-704e48066f0f_isrForCrosstalkSources_855820_28.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/acc25d0b-d59b-44b2-9bd3-704e48066f0f_isrForCrosstalkSources_855820_28.stderr parsl.dataflow.dflow INFO: Task 217 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca18196e44370c7ffb2100b33be86402 with result from task 217 parsl.dataflow.dflow DEBUG: Task 217 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/77c75561-2c65-4ae7-822d-aff25e4b8d4d_isrForCrosstalkSources_855821_59.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/77c75561-2c65-4ae7-822d-aff25e4b8d4d_isrForCrosstalkSources_855821_59.stderr parsl.dataflow.dflow INFO: Task 221 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0cf52b792cef1f56c7166ee5fa7c481a with result from task 221 parsl.dataflow.dflow DEBUG: Task 221 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/12cc124e-5722-4f7f-9fab-4615dec0c8e1_isrForCrosstalkSources_855911_53.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/12cc124e-5722-4f7f-9fab-4615dec0c8e1_isrForCrosstalkSources_855911_53.stderr parsl.dataflow.dflow INFO: Task 220 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43aef52b508aded26fdc2eefbf0908cb with result from task 220 parsl.dataflow.dflow DEBUG: Task 220 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/667d7035-29dc-4073-9c93-7a418399912c_isrForCrosstalkSources_855822_40.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/667d7035-29dc-4073-9c93-7a418399912c_isrForCrosstalkSources_855822_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:31 proc_lsst.multi:146 [INFO] found job 16968762 in provider astro proc_lsst.multi INFO: found job 16968762 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.dataflow.dflow INFO: Task 200 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a12fede80d5b2ab9bda24395ba858ff5 with result from task 200 parsl.dataflow.dflow DEBUG: Task 200 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/13ace5da-2534-436b-98a4-04467a6e5116_isrForCrosstalkSources_855822_34.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/13ace5da-2534-436b-98a4-04467a6e5116_isrForCrosstalkSources_855822_34.stderr parsl.dataflow.dflow INFO: Task 202 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2bdceb1dc15a106fd6ec58179161f47 with result from task 202 parsl.dataflow.dflow DEBUG: Task 202 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/397e2271-e1a6-45d2-9b8f-3b18d29daddc_isrForCrosstalkSources_855911_25.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/397e2271-e1a6-45d2-9b8f-3b18d29daddc_isrForCrosstalkSources_855911_25.stderr parsl.dataflow.dflow INFO: Task 195 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3fb1091897ee2fef69d81ba0e13347e0 with result from task 195 parsl.dataflow.dflow DEBUG: Task 195 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/60e9ad41-e577-46a3-bd15-22b54b9f2725_isrForCrosstalkSources_855910_25.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/60e9ad41-e577-46a3-bd15-22b54b9f2725_isrForCrosstalkSources_855910_25.stderr parsl.dataflow.dflow INFO: Task 192 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8b6d4d9b535902460d2133214c3fae2 with result from task 192 parsl.dataflow.dflow DEBUG: Task 192 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/29f7e352-7baa-4b0c-92f5-efd11679d3da_isrForCrosstalkSources_855820_13.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/29f7e352-7baa-4b0c-92f5-efd11679d3da_isrForCrosstalkSources_855820_13.stderr parsl.dataflow.dflow INFO: Task 203 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 146132281fd051bf44a0a25338179f9f with result from task 203 parsl.dataflow.dflow DEBUG: Task 203 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/3ebcb130-cd45-4ebb-acfc-0c4f486b8e93_isrForCrosstalkSources_855910_40.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/3ebcb130-cd45-4ebb-acfc-0c4f486b8e93_isrForCrosstalkSources_855910_40.stderr parsl.dataflow.dflow INFO: Task 205 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00802b939b4b836d41a1a38cbac31e53 with result from task 205 parsl.dataflow.dflow DEBUG: Task 205 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/6681672f-3839-459e-8d88-bc7d348aaf72_isrForCrosstalkSources_855909_39.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/6681672f-3839-459e-8d88-bc7d348aaf72_isrForCrosstalkSources_855909_39.stderr parsl.dataflow.dflow INFO: Task 204 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7320ee6031ac476d890abbd44a4726f0 with result from task 204 parsl.dataflow.dflow DEBUG: Task 204 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/32586c7f-741d-4c97-9b10-a4868dcf77b8_isrForCrosstalkSources_855911_12.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/32586c7f-741d-4c97-9b10-a4868dcf77b8_isrForCrosstalkSources_855911_12.stderr parsl.dataflow.dflow INFO: Task 194 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aaf8ab5d7e2b14049a4650bab33674f3 with result from task 194 parsl.dataflow.dflow DEBUG: Task 194 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/1b936b99-7e9a-4b76-a8f2-3b86cdce5093_isrForCrosstalkSources_855909_56.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/1b936b99-7e9a-4b76-a8f2-3b86cdce5093_isrForCrosstalkSources_855909_56.stderr parsl.dataflow.dflow INFO: Task 197 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b125b58ac9e57d56cfbb4b268dbae37f with result from task 197 parsl.dataflow.dflow DEBUG: Task 197 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/153c2c9a-2ba6-42a9-8757-357cc21aff11_isrForCrosstalkSources_855911_28.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/153c2c9a-2ba6-42a9-8757-357cc21aff11_isrForCrosstalkSources_855911_28.stderr parsl.dataflow.dflow INFO: Task 198 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2a3bca4fba7e5f8a8a8314a474bcc60 with result from task 198 parsl.dataflow.dflow DEBUG: Task 198 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/ca4fae14-b203-47a0-a6d5-7662d2ccf733_isrForCrosstalkSources_855910_35.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/ca4fae14-b203-47a0-a6d5-7662d2ccf733_isrForCrosstalkSources_855910_35.stderr parsl.dataflow.dflow INFO: Task 193 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 234623ad90897556b1c4d7d9c443deab with result from task 193 parsl.dataflow.dflow DEBUG: Task 193 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/2ad53253-bb4d-457c-99b9-26f013fb105b_isrForCrosstalkSources_855822_22.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/2ad53253-bb4d-457c-99b9-26f013fb105b_isrForCrosstalkSources_855822_22.stderr parsl.dataflow.dflow INFO: Task 206 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f8fa5c409cd046b8a006698041199d3 with result from task 206 parsl.dataflow.dflow DEBUG: Task 206 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/6280b9d8-5ad4-418e-91fe-de72f82997f8_isrForCrosstalkSources_855909_23.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/6280b9d8-5ad4-418e-91fe-de72f82997f8_isrForCrosstalkSources_855909_23.stderr parsl.dataflow.dflow INFO: Task 199 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d64253bb872811f9a09d7faad408b212 with result from task 199 parsl.dataflow.dflow DEBUG: Task 199 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/4c2a492e-a8c4-4ffa-9715-e68ab00f4490_isrForCrosstalkSources_855909_6.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/4c2a492e-a8c4-4ffa-9715-e68ab00f4490_isrForCrosstalkSources_855909_6.stderr parsl.dataflow.dflow INFO: Task 201 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44f5c7a7014171583d6de3b64efc7dd8 with result from task 201 parsl.dataflow.dflow DEBUG: Task 201 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/3e3484a8-8c02-4365-9bee-c5794880e13b_isrForCrosstalkSources_855820_1.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/3e3484a8-8c02-4365-9bee-c5794880e13b_isrForCrosstalkSources_855820_1.stderr parsl.dataflow.dflow INFO: Task 207 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b9cbb50a64d82461db40787895cc5fb with result from task 207 parsl.dataflow.dflow DEBUG: Task 207 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/5fdee1cb-7588-423f-9321-33dfb530e275_isrForCrosstalkSources_855911_62.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/5fdee1cb-7588-423f-9321-33dfb530e275_isrForCrosstalkSources_855911_62.stderr parsl.dataflow.dflow INFO: Task 196 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8f7f3b1a31b2e903e76c2d715c276fb with result from task 196 parsl.dataflow.dflow DEBUG: Task 196 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/fcc359f6-5bf4-4953-9ae3-c3ecb9409981_isrForCrosstalkSources_855909_49.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/fcc359f6-5bf4-4953-9ae3-c3ecb9409981_isrForCrosstalkSources_855909_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:33 proc_lsst.multi:146 [INFO] found job 16968763 in provider astro proc_lsst.multi INFO: found job 16968763 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:33 proc_lsst.multi:146 [INFO] found job 16968764 in provider astro proc_lsst.multi INFO: found job 16968764 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:33 proc_lsst.multi:146 [INFO] found job 16968765 in provider astro proc_lsst.multi INFO: found job 16968765 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:34 proc_lsst.multi:146 [INFO] found job 16968766 in provider astro proc_lsst.multi INFO: found job 16968766 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:34 proc_lsst.multi:146 [INFO] found job 16968767 in provider astro proc_lsst.multi INFO: found job 16968767 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:34 proc_lsst.multi:146 [INFO] found job 16968768 in provider astro proc_lsst.multi INFO: found job 16968768 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:35 proc_lsst.multi:146 [INFO] found job 16968769 in provider astro proc_lsst.multi INFO: found job 16968769 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:35 proc_lsst.multi:146 [INFO] found job 16968770 in provider astro proc_lsst.multi INFO: found job 16968770 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:35 proc_lsst.multi:146 [INFO] found job 16968771 in provider astro proc_lsst.multi INFO: found job 16968771 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:35 proc_lsst.multi:146 [INFO] found job 16968772 in provider astro proc_lsst.multi INFO: found job 16968772 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:35 proc_lsst.multi:146 [INFO] found job 16968773 in provider astro proc_lsst.multi INFO: found job 16968773 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:35 proc_lsst.multi:146 [INFO] found job 16968774 in provider astro proc_lsst.multi INFO: found job 16968774 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.dataflow.dflow INFO: Task 332 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 255c0022a01b08a688663a28dc0b016a with result from task 332 parsl.dataflow.dflow DEBUG: Task 332 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/72e5b858-e552-466d-86a4-015f6939f399_isrForCrosstalkSources_855822_25.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/72e5b858-e552-466d-86a4-015f6939f399_isrForCrosstalkSources_855822_25.stderr parsl.dataflow.dflow INFO: Task 269 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 063759d5639aa7f57ec0d196ede0d147 with result from task 269 parsl.dataflow.dflow DEBUG: Task 269 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/f24f9c17-b202-4416-a26a-f27644492ce7_isrForCrosstalkSources_855910_26.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/f24f9c17-b202-4416-a26a-f27644492ce7_isrForCrosstalkSources_855910_26.stderr parsl.dataflow.dflow INFO: Task 339 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 807555cacd8fa55588f25abd8212939f with result from task 339 parsl.dataflow.dflow DEBUG: Task 339 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/f9f30cef-d7ca-4a8c-a181-be8ad0075aaa_isrForCrosstalkSources_855911_19.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/f9f30cef-d7ca-4a8c-a181-be8ad0075aaa_isrForCrosstalkSources_855911_19.stderr parsl.dataflow.dflow INFO: Task 337 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 633b694fe653ea1146140ca7c203242e with result from task 337 parsl.dataflow.dflow DEBUG: Task 337 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/c8d7be2d-699a-41e5-a2fa-207eefdcffe6_isrForCrosstalkSources_855911_39.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/c8d7be2d-699a-41e5-a2fa-207eefdcffe6_isrForCrosstalkSources_855911_39.stderr parsl.dataflow.dflow INFO: Task 326 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3537ceee33aa614472d7376940f431e8 with result from task 326 parsl.dataflow.dflow DEBUG: Task 326 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/08556b91-46f2-4336-9838-1cf9dc515561_isrForCrosstalkSources_855822_35.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/08556b91-46f2-4336-9838-1cf9dc515561_isrForCrosstalkSources_855822_35.stderr parsl.dataflow.dflow INFO: Task 330 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e94b715f44fa6208056422739a18ede with result from task 330 parsl.dataflow.dflow DEBUG: Task 330 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/0983558b-06a1-4785-809b-c9bbc4e392ef_isrForCrosstalkSources_855821_24.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/0983558b-06a1-4785-809b-c9bbc4e392ef_isrForCrosstalkSources_855821_24.stderr parsl.dataflow.dflow INFO: Task 338 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60b49f8e570dc23cf6af7782f0a6df50 with result from task 338 parsl.dataflow.dflow DEBUG: Task 338 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/7544a6e0-4384-4601-b5dc-f7cb019dff1c_isrForCrosstalkSources_855821_8.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/7544a6e0-4384-4601-b5dc-f7cb019dff1c_isrForCrosstalkSources_855821_8.stderr parsl.dataflow.dflow INFO: Task 283 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b3c098d7fed52cc3555a8778d54be6a with result from task 283 parsl.dataflow.dflow DEBUG: Task 283 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/3a1a6172-5a90-4b08-9fff-e859baa02dc2_isrForCrosstalkSources_855911_17.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/3a1a6172-5a90-4b08-9fff-e859baa02dc2_isrForCrosstalkSources_855911_17.stderr parsl.dataflow.dflow INFO: Task 279 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5d29a4f360d2f9075424bea744914e7 with result from task 279 parsl.dataflow.dflow DEBUG: Task 279 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/1201a4e6-5ab4-4211-880c-5db79910c576_isrForCrosstalkSources_855911_23.stdout parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/1201a4e6-5ab4-4211-880c-5db79910c576_isrForCrosstalkSources_855911_23.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:36 proc_lsst.multi:146 [INFO] found job 16968775 in provider astro proc_lsst.multi INFO: found job 16968775 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.dataflow.dflow INFO: Task 340 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3999abb5388952a2ceec50f7bdfbce47 with result from task 340 parsl.dataflow.dflow DEBUG: Task 340 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/5ba133c1-7da1-480a-b0e8-fe2b62927ee5_isrForCrosstalkSources_855820_10.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/5ba133c1-7da1-480a-b0e8-fe2b62927ee5_isrForCrosstalkSources_855820_10.stderr parsl.dataflow.dflow INFO: Task 278 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7cf84fedbc0561a9bcfb5b266230ef51 with result from task 278 parsl.dataflow.dflow DEBUG: Task 278 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/f773b98e-dd0c-4976-a583-f83bdf4bb20f_isrForCrosstalkSources_855910_15.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/f773b98e-dd0c-4976-a583-f83bdf4bb20f_isrForCrosstalkSources_855910_15.stderr parsl.dataflow.dflow INFO: Task 280 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a181b23a81211cbabe4368f39c67e94 with result from task 280 parsl.dataflow.dflow DEBUG: Task 280 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/4ab750c2-2134-4425-b3c2-f12bf2420cca_isrForCrosstalkSources_855820_15.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/4ab750c2-2134-4425-b3c2-f12bf2420cca_isrForCrosstalkSources_855820_15.stderr parsl.dataflow.dflow INFO: Task 333 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4db25eb0995d6f689eb54cda9d39bbf6 with result from task 333 parsl.dataflow.dflow DEBUG: Task 333 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/3132d585-94a2-4c71-89b1-add95b35a38d_isrForCrosstalkSources_855821_40.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/3132d585-94a2-4c71-89b1-add95b35a38d_isrForCrosstalkSources_855821_40.stderr parsl.dataflow.dflow INFO: Task 328 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87fcc7444a5420ebf9b6705fff446524 with result from task 328 parsl.dataflow.dflow DEBUG: Task 328 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/c4b90971-b144-4d43-b439-b4bc61cc84cf_isrForCrosstalkSources_855910_29.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/c4b90971-b144-4d43-b439-b4bc61cc84cf_isrForCrosstalkSources_855910_29.stderr parsl.dataflow.dflow INFO: Task 325 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d31482d10a58b8ef59d749e16bfb31d4 with result from task 325 parsl.dataflow.dflow DEBUG: Task 325 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/7deb925d-4e10-4e82-be32-82f55af462e1_isrForCrosstalkSources_855820_48.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/7deb925d-4e10-4e82-be32-82f55af462e1_isrForCrosstalkSources_855820_48.stderr parsl.dataflow.dflow INFO: Task 331 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56b0f272e216781294ebd31779b328cd with result from task 331 parsl.dataflow.dflow DEBUG: Task 331 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/b679e40d-79e4-4db4-ab4a-4854c758914e_isrForCrosstalkSources_855820_54.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/b679e40d-79e4-4db4-ab4a-4854c758914e_isrForCrosstalkSources_855820_54.stderr parsl.dataflow.dflow INFO: Task 329 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bbc85fc2fb4c0f35e39f13d17f7601ab with result from task 329 parsl.dataflow.dflow DEBUG: Task 329 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/cb46f9d6-88c4-477f-bdbc-b0e912c1a2ee_isrForCrosstalkSources_855909_38.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/cb46f9d6-88c4-477f-bdbc-b0e912c1a2ee_isrForCrosstalkSources_855909_38.stderr parsl.dataflow.dflow INFO: Task 277 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d592f4ad700606f485abb90d4c95be08 with result from task 277 parsl.dataflow.dflow DEBUG: Task 277 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/ef3fb890-b96a-4580-92b9-3b0cd74379c2_isrForCrosstalkSources_855909_45.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/ef3fb890-b96a-4580-92b9-3b0cd74379c2_isrForCrosstalkSources_855909_45.stderr parsl.dataflow.dflow INFO: Task 327 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a9aaf9b709ca99080699a670b6559e3 with result from task 327 parsl.dataflow.dflow DEBUG: Task 327 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/2604833f-a121-49ab-ae73-03440f21a1b3_isrForCrosstalkSources_855820_59.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/2604833f-a121-49ab-ae73-03440f21a1b3_isrForCrosstalkSources_855820_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:36 proc_lsst.multi:146 [INFO] found job 16968776 in provider astro proc_lsst.multi INFO: found job 16968776 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.dataflow.dflow INFO: Task 276 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d588906a261f9a1ae0f95ecdbfa3154 with result from task 276 parsl.dataflow.dflow DEBUG: Task 276 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/b1faafd2-5206-4c80-a812-245cdf1061ce_isrForCrosstalkSources_855911_3.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/b1faafd2-5206-4c80-a812-245cdf1061ce_isrForCrosstalkSources_855911_3.stderr parsl.dataflow.dflow INFO: Task 281 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3b460ffc085e90a501c5af03b5fc417 with result from task 281 parsl.dataflow.dflow DEBUG: Task 281 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/7e540e51-292b-4c64-a422-86035fa212cd_isrForCrosstalkSources_855910_1.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/7e540e51-292b-4c64-a422-86035fa212cd_isrForCrosstalkSources_855910_1.stderr parsl.dataflow.dflow INFO: Task 335 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d262d45c2443ba035e298f76630acde4 with result from task 335 parsl.dataflow.dflow DEBUG: Task 335 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/1e6b0344-25a8-4fac-ad18-d833931319ca_isrForCrosstalkSources_855911_58.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/1e6b0344-25a8-4fac-ad18-d833931319ca_isrForCrosstalkSources_855911_58.stderr parsl.dataflow.dflow INFO: Task 271 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f940ffe5d331c77441abc9ae954db895 with result from task 271 parsl.dataflow.dflow DEBUG: Task 271 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/e6a2934d-a52e-4d8b-b913-3b6dee1ad150_isrForCrosstalkSources_855911_44.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/e6a2934d-a52e-4d8b-b913-3b6dee1ad150_isrForCrosstalkSources_855911_44.stderr parsl.dataflow.dflow INFO: Task 275 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 730c99d63433688a19005a99874b925a with result from task 275 parsl.dataflow.dflow DEBUG: Task 275 checkpointed parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/2aa18c98-1195-4d3c-bcf2-cd188c95f409_isrForCrosstalkSources_855910_30.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/2aa18c98-1195-4d3c-bcf2-cd188c95f409_isrForCrosstalkSources_855910_30.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:36 proc_lsst.multi:146 [INFO] found job 16968777 in provider astro proc_lsst.multi INFO: found job 16968777 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.dataflow.dflow INFO: Task 282 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4851630223bce9cb78b4f540a3015c03 with result from task 282 parsl.dataflow.dflow DEBUG: Task 282 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/d4087a6b-d3dd-4d72-8032-0b228dfac24a_isrForCrosstalkSources_855821_7.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/d4087a6b-d3dd-4d72-8032-0b228dfac24a_isrForCrosstalkSources_855821_7.stderr parsl.dataflow.dflow INFO: Task 270 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 338e2bcc22edeb5d43bcc2765dc5293c with result from task 270 parsl.dataflow.dflow DEBUG: Task 270 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/eddbd2d3-4c81-482e-b84f-1a4ba689bc36_isrForCrosstalkSources_855911_43.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/eddbd2d3-4c81-482e-b84f-1a4ba689bc36_isrForCrosstalkSources_855911_43.stderr parsl.dataflow.dflow INFO: Task 272 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc227710b4c67670939f3cb3fef8834c with result from task 272 parsl.dataflow.dflow DEBUG: Task 272 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/103327fd-551d-403c-b6a1-d51897922402_isrForCrosstalkSources_855822_57.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/103327fd-551d-403c-b6a1-d51897922402_isrForCrosstalkSources_855822_57.stderr parsl.dataflow.dflow INFO: Task 273 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a992c16a3800ca83b3b2f0e9b7bc1f1 with result from task 273 parsl.dataflow.dflow DEBUG: Task 273 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/5a6ec4e4-3b55-47a6-948b-00b62bbc90a9_isrForCrosstalkSources_855909_58.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/5a6ec4e4-3b55-47a6-948b-00b62bbc90a9_isrForCrosstalkSources_855909_58.stderr parsl.dataflow.dflow INFO: Task 336 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6a1bcf5ad7351dba216258d193ebaf1 with result from task 336 parsl.dataflow.dflow DEBUG: Task 336 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/65085679-9ca8-4b78-9053-3ccca513cb66_isrForCrosstalkSources_855909_35.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/65085679-9ca8-4b78-9053-3ccca513cb66_isrForCrosstalkSources_855909_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Task 334 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.memoization DEBUG: Storing app cache entry 2ec5cebe69d8b0c80bb5917f62b2f467 with result from task 334 parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow DEBUG: Task 334 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/9c824a62-331f-45fa-8fe0-8a0ddd5c9f2e_isrForCrosstalkSources_855820_43.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/9c824a62-331f-45fa-8fe0-8a0ddd5c9f2e_isrForCrosstalkSources_855820_43.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:36 proc_lsst.multi:146 [INFO] found job 16968778 in provider astro proc_lsst.multi INFO: found job 16968778 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.dataflow.dflow INFO: Task 284 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ada684595c53eb9495844296cbd04bf0 with result from task 284 parsl.dataflow.dflow DEBUG: Task 284 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/ee703511-563b-447a-a833-a761c2037225_isrForCrosstalkSources_855909_62.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/ee703511-563b-447a-a833-a761c2037225_isrForCrosstalkSources_855909_62.stderr parsl.dataflow.dflow INFO: Task 274 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b555937498fb4c6d1e094ae245ec290 with result from task 274 parsl.dataflow.dflow DEBUG: Task 274 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/3e0bfff9-0a83-436d-8efb-962177ee6a7c_isrForCrosstalkSources_855820_58.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/3e0bfff9-0a83-436d-8efb-962177ee6a7c_isrForCrosstalkSources_855820_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:36 proc_lsst.multi:146 [INFO] found job 16968779 in provider astro proc_lsst.multi INFO: found job 16968779 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:37 proc_lsst.multi:146 [INFO] found job 16968780 in provider escience_rtx6k proc_lsst.multi INFO: found job 16968780 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968780,16968781,16968782,16968783,16968784' parsl.providers.slurm.slurm DEBUG: squeue returned 16968781 R 16968782 R 16968783 R 16968784 R 16968780 R parsl.providers.slurm.slurm DEBUG: Updating job 16968781 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968780 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:37 proc_lsst.multi:146 [INFO] found job 16968781 in provider escience_rtx6k proc_lsst.multi INFO: found job 16968781 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968780,16968781,16968782,16968783,16968784' parsl.providers.slurm.slurm DEBUG: squeue returned 16968781 R 16968782 R 16968783 R 16968784 R 16968780 R parsl.providers.slurm.slurm DEBUG: Updating job 16968781 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968780 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:37 proc_lsst.multi:146 [INFO] found job 16968782 in provider escience_rtx6k proc_lsst.multi INFO: found job 16968782 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968780,16968781,16968782,16968783,16968784' parsl.providers.slurm.slurm DEBUG: squeue returned 16968781 R 16968782 R 16968783 R 16968784 R 16968780 R parsl.providers.slurm.slurm DEBUG: Updating job 16968781 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968780 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:37 proc_lsst.multi:146 [INFO] found job 16968783 in provider escience_rtx6k proc_lsst.multi INFO: found job 16968783 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968780,16968781,16968782,16968783,16968784' parsl.providers.slurm.slurm DEBUG: squeue returned 16968781 R 16968782 R 16968783 R 16968784 R 16968780 R parsl.providers.slurm.slurm DEBUG: Updating job 16968781 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968780 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:37 proc_lsst.multi:146 [INFO] found job 16968784 in provider escience_rtx6k proc_lsst.multi INFO: found job 16968784 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968780,16968781,16968782,16968783,16968784' parsl.providers.slurm.slurm DEBUG: squeue returned 16968781 R 16968782 R 16968783 R 16968784 R 16968780 R parsl.providers.slurm.slurm DEBUG: Updating job 16968781 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968780 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:38 proc_lsst.multi:146 [INFO] found job 16968785 in provider escience_a40 proc_lsst.multi INFO: found job 16968785 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:38 proc_lsst.multi:146 [INFO] found job 16968786 in provider escience_a40 proc_lsst.multi INFO: found job 16968786 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:38 proc_lsst.multi:146 [INFO] found job 16968787 in provider escience_a40 proc_lsst.multi INFO: found job 16968787 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:38 proc_lsst.multi:146 [INFO] found job 16968788 in provider escience_a40 proc_lsst.multi INFO: found job 16968788 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:38 proc_lsst.multi:146 [INFO] found job 16968789 in provider escience_a40 proc_lsst.multi INFO: found job 16968789 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:39 proc_lsst.multi:146 [INFO] found job 16968790 in provider escience_a40 proc_lsst.multi INFO: found job 16968790 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:39 proc_lsst.multi:146 [INFO] found job 16968791 in provider escience_a40 proc_lsst.multi INFO: found job 16968791 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:39 proc_lsst.multi:146 [INFO] found job 16968792 in provider escience_a40 proc_lsst.multi INFO: found job 16968792 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:39 proc_lsst.multi:146 [INFO] found job 16968793 in provider escience_a40 proc_lsst.multi INFO: found job 16968793 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:39 proc_lsst.multi:146 [INFO] found job 16968794 in provider escience_a40 proc_lsst.multi INFO: found job 16968794 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:40 proc_lsst.multi:146 [INFO] found job 16968795 in provider escience_a40 proc_lsst.multi INFO: found job 16968795 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:40 proc_lsst.multi:146 [INFO] found job 16968796 in provider escience_a40 proc_lsst.multi INFO: found job 16968796 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:40 proc_lsst.multi:146 [INFO] found job 16968797 in provider escience_a40 proc_lsst.multi INFO: found job 16968797 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:40 proc_lsst.multi:146 [INFO] found job 16968798 in provider escience_a40 proc_lsst.multi INFO: found job 16968798 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:40 proc_lsst.multi:146 [INFO] found job 16968799 in provider escience_a40 proc_lsst.multi INFO: found job 16968799 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:41 proc_lsst.multi:146 [INFO] found job 16968800 in provider escience_a40 proc_lsst.multi INFO: found job 16968800 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.dataflow.dflow INFO: Task 371 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa7ca770f5c706c23391079f041da27f with result from task 371 parsl.dataflow.dflow DEBUG: Task 371 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/f34e34c9-0ca4-475a-864e-67138acbd4ed_isrForCrosstalkSources_855821_10.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/f34e34c9-0ca4-475a-864e-67138acbd4ed_isrForCrosstalkSources_855821_10.stderr parsl.dataflow.dflow INFO: Task 368 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28f69254e4d4f4d3445f815abd7df41f with result from task 368 parsl.dataflow.dflow DEBUG: Task 368 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/0b37fdd1-59b1-4bb0-bc1d-149e98b10252_isrForCrosstalkSources_855909_8.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/0b37fdd1-59b1-4bb0-bc1d-149e98b10252_isrForCrosstalkSources_855909_8.stderr parsl.dataflow.dflow INFO: Task 369 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d753b5243ff7d207129cb46834ffa397 with result from task 369 parsl.dataflow.dflow DEBUG: Task 369 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/eb821e3c-4f02-4ce8-9f2c-1d159bae605d_isrForCrosstalkSources_855820_14.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/eb821e3c-4f02-4ce8-9f2c-1d159bae605d_isrForCrosstalkSources_855820_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:41 proc_lsst.multi:146 [INFO] found job 16968805 in provider escience_a40 proc_lsst.multi INFO: found job 16968805 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.dataflow.dflow INFO: Task 367 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c933ca5fa1357f76299e3b628508c837 with result from task 367 parsl.dataflow.dflow DEBUG: Task 367 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/a750bce9-13f4-4b4d-92fb-a6e8168c2540_isrForCrosstalkSources_855910_46.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/a750bce9-13f4-4b4d-92fb-a6e8168c2540_isrForCrosstalkSources_855910_46.stderr parsl.dataflow.dflow INFO: Task 370 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2668468b069cc4ec9d75ec72d68a07e3 with result from task 370 parsl.dataflow.dflow DEBUG: Task 370 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/fbfd239d-40f2-4696-8b81-f66ee81f2af0_isrForCrosstalkSources_855910_13.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/fbfd239d-40f2-4696-8b81-f66ee81f2af0_isrForCrosstalkSources_855910_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:41 proc_lsst.multi:146 [INFO] found job 16968811 in provider escience_a40 proc_lsst.multi INFO: found job 16968811 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.dataflow.dflow INFO: Task 362 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a033bcdb24f08c085d5fca97b412192e with result from task 362 parsl.dataflow.dflow DEBUG: Task 362 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/7143b02f-f9e6-44e5-b3f2-7236f9726871_isrForCrosstalkSources_855910_19.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/7143b02f-f9e6-44e5-b3f2-7236f9726871_isrForCrosstalkSources_855910_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:41 proc_lsst.multi:146 [INFO] found job 16968812 in provider escience_a40 proc_lsst.multi INFO: found job 16968812 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.dataflow.dflow INFO: Task 69 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f19a346fe9dd6dbd859c6ca65f0cc2d7 with result from task 69 parsl.dataflow.dflow DEBUG: Task 69 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/82e36f6e-b7cc-4ae3-b67b-15d6ebf09227_isrForCrosstalkSources_855909_54.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/82e36f6e-b7cc-4ae3-b67b-15d6ebf09227_isrForCrosstalkSources_855909_54.stderr parsl.dataflow.dflow INFO: Task 361 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0db9bd2e84dcc55178c9bda6fb5c7db9 with result from task 361 parsl.dataflow.dflow DEBUG: Task 361 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/7d92a25b-f402-471d-90dc-dfca83c2ce9c_isrForCrosstalkSources_855822_10.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/7d92a25b-f402-471d-90dc-dfca83c2ce9c_isrForCrosstalkSources_855822_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:41 proc_lsst.multi:146 [INFO] found job 16968813 in provider escience_a40 proc_lsst.multi INFO: found job 16968813 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.dataflow.dflow INFO: Task 347 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83b1a59621a8077a72f72dc89d89cc7b with result from task 347 parsl.dataflow.dflow DEBUG: Task 347 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/0f4f52a1-eb9e-4192-b920-a26da581eada_isrForCrosstalkSources_855821_42.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/0f4f52a1-eb9e-4192-b920-a26da581eada_isrForCrosstalkSources_855821_42.stderr parsl.dataflow.dflow INFO: Task 349 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c2568c620e3b6d92b734e7e628d0a3f with result from task 349 parsl.dataflow.dflow DEBUG: Task 349 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/baa7d245-f436-44ab-83f6-ece0d486e9f0_isrForCrosstalkSources_855909_13.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/baa7d245-f436-44ab-83f6-ece0d486e9f0_isrForCrosstalkSources_855909_13.stderr parsl.dataflow.dflow INFO: Task 345 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c773a06779c81bb16e690ad34ef70dcf with result from task 345 parsl.dataflow.dflow DEBUG: Task 345 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/e45b6946-39cd-43ac-b171-cef8296b49d1_isrForCrosstalkSources_855911_55.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/e45b6946-39cd-43ac-b171-cef8296b49d1_isrForCrosstalkSources_855911_55.stderr parsl.dataflow.dflow INFO: Task 359 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d82b6e3e9f2a92e561327c48f329062f with result from task 359 parsl.dataflow.dflow DEBUG: Task 359 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/13979976-074d-4b3e-bada-af4b0f8575aa_isrForCrosstalkSources_855820_45.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/13979976-074d-4b3e-bada-af4b0f8575aa_isrForCrosstalkSources_855820_45.stderr parsl.dataflow.dflow INFO: Task 366 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29dcc586fbd47f0bced4e825093628c5 with result from task 366 parsl.dataflow.dflow DEBUG: Task 366 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/66ac3c44-f514-43c5-8373-06f4e94fb18c_isrForCrosstalkSources_855911_57.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/66ac3c44-f514-43c5-8373-06f4e94fb18c_isrForCrosstalkSources_855911_57.stderr parsl.dataflow.dflow INFO: Task 360 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d257a47e973fc1f057953fd281bd21af with result from task 360 parsl.dataflow.dflow DEBUG: Task 360 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/bda9872a-8638-432c-a37f-68dd76105613_isrForCrosstalkSources_855822_9.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/bda9872a-8638-432c-a37f-68dd76105613_isrForCrosstalkSources_855822_9.stderr parsl.dataflow.dflow INFO: Task 365 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d0473a77821e93833c0631c555f51b2 with result from task 365 parsl.dataflow.dflow DEBUG: Task 365 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/0174a0f7-4502-4861-ade3-359921b53cb5_isrForCrosstalkSources_855822_49.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/0174a0f7-4502-4861-ade3-359921b53cb5_isrForCrosstalkSources_855822_49.stderr parsl.dataflow.dflow INFO: Task 350 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83f882c7cc9944491d0418b58def5bb9 with result from task 350 parsl.dataflow.dflow DEBUG: Task 350 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/b18c4e06-b70b-46ea-9b66-b4dc5ea65000_isrForCrosstalkSources_855822_48.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/b18c4e06-b70b-46ea-9b66-b4dc5ea65000_isrForCrosstalkSources_855822_48.stderr parsl.dataflow.dflow INFO: Task 346 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39ebd695e7410ab377fac3d52a3d1347 with result from task 346 parsl.dataflow.dflow DEBUG: Task 346 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/69d66858-f93e-4a2f-b7f6-2683e82d95da_isrForCrosstalkSources_855820_32.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/69d66858-f93e-4a2f-b7f6-2683e82d95da_isrForCrosstalkSources_855820_32.stderr parsl.dataflow.dflow INFO: Task 344 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89d05f23fe238502a5c5ac30adc096d5 with result from task 344 parsl.dataflow.dflow DEBUG: Task 344 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/452b1d6e-6fa4-4564-8d75-35dd391d0438_isrForCrosstalkSources_855822_27.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/452b1d6e-6fa4-4564-8d75-35dd391d0438_isrForCrosstalkSources_855822_27.stderr parsl.dataflow.dflow INFO: Task 364 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d839f63a05dcbf23e7a5f018def9af94 with result from task 364 parsl.dataflow.dflow DEBUG: Task 364 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/4f2c09f2-6021-4e18-9fb6-b07ec33e06e7_isrForCrosstalkSources_855909_57.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/4f2c09f2-6021-4e18-9fb6-b07ec33e06e7_isrForCrosstalkSources_855909_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:42 proc_lsst.multi:146 [INFO] found job 16968814 in provider checkpoint proc_lsst.multi INFO: found job 16968814 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.dataflow.dflow INFO: Task 348 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60a054ef23811d640702a86915c2a74d with result from task 348 parsl.dataflow.dflow DEBUG: Task 348 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/6214c10f-5f4a-411d-a1d4-2b2284ce4e72_isrForCrosstalkSources_855821_60.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/6214c10f-5f4a-411d-a1d4-2b2284ce4e72_isrForCrosstalkSources_855821_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:42 proc_lsst.multi:146 [INFO] found job 16968815 in provider checkpoint proc_lsst.multi INFO: found job 16968815 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.dataflow.dflow INFO: Task 363 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 572d54305704cb6da33a1204a04323ab with result from task 363 parsl.dataflow.dflow DEBUG: Task 363 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/8bd21cef-1bbf-4b58-b3d8-bc3e66e044d6_isrForCrosstalkSources_855820_55.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/8bd21cef-1bbf-4b58-b3d8-bc3e66e044d6_isrForCrosstalkSources_855820_55.stderr parsl.dataflow.dflow INFO: Task 343 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5cdd30fd04c9daf0fdf60569796f45be with result from task 343 parsl.dataflow.dflow DEBUG: Task 343 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/6955dac6-f691-4a52-aba4-9a3ffaefbcbd_isrForCrosstalkSources_855910_50.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/6955dac6-f691-4a52-aba4-9a3ffaefbcbd_isrForCrosstalkSources_855910_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:42 proc_lsst.multi:146 [INFO] found job 16968816 in provider checkpoint proc_lsst.multi INFO: found job 16968816 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:42 proc_lsst.multi:146 [INFO] found job 16968817 in provider checkpoint proc_lsst.multi INFO: found job 16968817 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:42 proc_lsst.multi:146 [INFO] found job 16968818 in provider checkpoint proc_lsst.multi INFO: found job 16968818 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:43 proc_lsst.multi:146 [INFO] found job 16968819 in provider checkpoint proc_lsst.multi INFO: found job 16968819 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:43 proc_lsst.multi:146 [INFO] found job 16968823 in provider checkpoint proc_lsst.multi INFO: found job 16968823 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:43 proc_lsst.multi:146 [INFO] found job 16968824 in provider checkpoint proc_lsst.multi INFO: found job 16968824 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:43 proc_lsst.multi:146 [INFO] found job 16968825 in provider checkpoint proc_lsst.multi INFO: found job 16968825 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:43 proc_lsst.multi:146 [INFO] found job 16968826 in provider checkpoint proc_lsst.multi INFO: found job 16968826 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:44 proc_lsst.multi:146 [INFO] found job 16968827 in provider checkpoint proc_lsst.multi INFO: found job 16968827 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:44 proc_lsst.multi:146 [INFO] found job 16968828 in provider checkpoint proc_lsst.multi INFO: found job 16968828 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:44 proc_lsst.multi:146 [INFO] found job 16968829 in provider checkpoint proc_lsst.multi INFO: found job 16968829 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:44 proc_lsst.multi:146 [INFO] found job 16968830 in provider checkpoint proc_lsst.multi INFO: found job 16968830 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:44 proc_lsst.multi:146 [INFO] found job 16968831 in provider checkpoint proc_lsst.multi INFO: found job 16968831 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:45 proc_lsst.multi:146 [INFO] found job 16968832 in provider checkpoint proc_lsst.multi INFO: found job 16968832 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:45 proc_lsst.multi:146 [INFO] found job 16968833 in provider checkpoint proc_lsst.multi INFO: found job 16968833 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:45 proc_lsst.multi:146 [INFO] found job 16968834 in provider checkpoint proc_lsst.multi INFO: found job 16968834 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:45 proc_lsst.multi:146 [INFO] found job 16968835 in provider checkpoint proc_lsst.multi INFO: found job 16968835 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:45 proc_lsst.multi:146 [INFO] found job 16968836 in provider checkpoint proc_lsst.multi INFO: found job 16968836 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:46 proc_lsst.multi:146 [INFO] found job 16968837 in provider checkpoint proc_lsst.multi INFO: found job 16968837 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:46 proc_lsst.multi:146 [INFO] found job 16968838 in provider checkpoint proc_lsst.multi INFO: found job 16968838 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:46 proc_lsst.multi:146 [INFO] found job 16968839 in provider checkpoint proc_lsst.multi INFO: found job 16968839 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:46 proc_lsst.multi:146 [INFO] found job 16968840 in provider checkpoint proc_lsst.multi INFO: found job 16968840 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:46 proc_lsst.multi:146 [INFO] found job 16968841 in provider checkpoint proc_lsst.multi INFO: found job 16968841 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:47 proc_lsst.multi:146 [INFO] found job 16968842 in provider checkpoint proc_lsst.multi INFO: found job 16968842 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:47 proc_lsst.multi:146 [INFO] found job 16968843 in provider checkpoint proc_lsst.multi INFO: found job 16968843 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:47 proc_lsst.multi:146 [INFO] found job 16968844 in provider checkpoint proc_lsst.multi INFO: found job 16968844 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:47 proc_lsst.multi:146 [INFO] found job 16968845 in provider checkpoint proc_lsst.multi INFO: found job 16968845 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:47 proc_lsst.multi:146 [INFO] found job 16968846 in provider checkpoint proc_lsst.multi INFO: found job 16968846 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:48 proc_lsst.multi:146 [INFO] found job 16968847 in provider checkpoint proc_lsst.multi INFO: found job 16968847 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:48 proc_lsst.multi:146 [INFO] found job 16968848 in provider checkpoint proc_lsst.multi INFO: found job 16968848 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:48 proc_lsst.multi:146 [INFO] found job 16968849 in provider checkpoint proc_lsst.multi INFO: found job 16968849 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:48 proc_lsst.multi:146 [INFO] found job 16968850 in provider checkpoint proc_lsst.multi INFO: found job 16968850 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:48 proc_lsst.multi:146 [INFO] found job 16968851 in provider checkpoint proc_lsst.multi INFO: found job 16968851 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:49 proc_lsst.multi:146 [INFO] found job 16968852 in provider checkpoint proc_lsst.multi INFO: found job 16968852 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:49 proc_lsst.multi:146 [INFO] found job 16968853 in provider checkpoint proc_lsst.multi INFO: found job 16968853 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 40 parsl.dataflow.strategy DEBUG: Executor multi has 40 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22762785682704 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 40 parsl.dataflow.strategy DEBUG: Executor multi has 40 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22762785682704 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 40 parsl.dataflow.strategy DEBUG: Executor multi has 40 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22762785682704 2024-03-09 20:51:59 proc_lsst.multi:146 [INFO] found job 35332 in provider local proc_lsst.multi INFO: found job 35332 in provider local 2024-03-09 20:51:59 proc_lsst.multi:146 [INFO] found job 16968750 in provider astro proc_lsst.multi INFO: found job 16968750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:59 proc_lsst.multi:146 [INFO] found job 16968751 in provider astro proc_lsst.multi INFO: found job 16968751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:51:59 proc_lsst.multi:146 [INFO] found job 16968752 in provider astro proc_lsst.multi INFO: found job 16968752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:00 proc_lsst.multi:146 [INFO] found job 16968753 in provider astro proc_lsst.multi INFO: found job 16968753 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:00 proc_lsst.multi:146 [INFO] found job 16968754 in provider astro proc_lsst.multi INFO: found job 16968754 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:00 proc_lsst.multi:146 [INFO] found job 16968755 in provider astro proc_lsst.multi INFO: found job 16968755 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:00 proc_lsst.multi:146 [INFO] found job 16968756 in provider astro proc_lsst.multi INFO: found job 16968756 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:00 proc_lsst.multi:146 [INFO] found job 16968757 in provider astro proc_lsst.multi INFO: found job 16968757 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:01 proc_lsst.multi:146 [INFO] found job 16968758 in provider astro proc_lsst.multi INFO: found job 16968758 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:01 proc_lsst.multi:146 [INFO] found job 16968759 in provider astro proc_lsst.multi INFO: found job 16968759 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:01 proc_lsst.multi:146 [INFO] found job 16968760 in provider astro proc_lsst.multi INFO: found job 16968760 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:01 proc_lsst.multi:146 [INFO] found job 16968761 in provider astro proc_lsst.multi INFO: found job 16968761 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:01 proc_lsst.multi:146 [INFO] found job 16968762 in provider astro proc_lsst.multi INFO: found job 16968762 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:02 proc_lsst.multi:146 [INFO] found job 16968763 in provider astro proc_lsst.multi INFO: found job 16968763 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:02 proc_lsst.multi:146 [INFO] found job 16968764 in provider astro proc_lsst.multi INFO: found job 16968764 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:02 proc_lsst.multi:146 [INFO] found job 16968765 in provider astro proc_lsst.multi INFO: found job 16968765 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:02 proc_lsst.multi:146 [INFO] found job 16968766 in provider astro proc_lsst.multi INFO: found job 16968766 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:02 proc_lsst.multi:146 [INFO] found job 16968767 in provider astro proc_lsst.multi INFO: found job 16968767 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:03 proc_lsst.multi:146 [INFO] found job 16968768 in provider astro proc_lsst.multi INFO: found job 16968768 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:03 proc_lsst.multi:146 [INFO] found job 16968769 in provider astro proc_lsst.multi INFO: found job 16968769 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:03 proc_lsst.multi:146 [INFO] found job 16968770 in provider astro proc_lsst.multi INFO: found job 16968770 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:03 proc_lsst.multi:146 [INFO] found job 16968771 in provider astro proc_lsst.multi INFO: found job 16968771 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:03 proc_lsst.multi:146 [INFO] found job 16968772 in provider astro proc_lsst.multi INFO: found job 16968772 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:04 proc_lsst.multi:146 [INFO] found job 16968773 in provider astro proc_lsst.multi INFO: found job 16968773 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:04 proc_lsst.multi:146 [INFO] found job 16968774 in provider astro proc_lsst.multi INFO: found job 16968774 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:06 proc_lsst.multi:146 [INFO] found job 16968775 in provider astro proc_lsst.multi INFO: found job 16968775 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:06 proc_lsst.multi:146 [INFO] found job 16968776 in provider astro proc_lsst.multi INFO: found job 16968776 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:06 proc_lsst.multi:146 [INFO] found job 16968777 in provider astro proc_lsst.multi INFO: found job 16968777 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:07 proc_lsst.multi:146 [INFO] found job 16968778 in provider astro proc_lsst.multi INFO: found job 16968778 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:07 proc_lsst.multi:146 [INFO] found job 16968779 in provider astro proc_lsst.multi INFO: found job 16968779 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968750,16968751,16968752,16968753,16968754,16968755,16968756,16968757,16968758,16968759,16968760,16968761,16968762,16968763,16968764,16968765,16968766,16968767,16968768,16968769,16968770,16968771,16968772,16968773,16968774,16968775,16968776,16968777,16968778,16968779' parsl.providers.slurm.slurm DEBUG: squeue returned 16968779 R 16968773 R 16968774 R 16968775 R 16968776 R 16968777 R 16968778 R 16968766 R 16968767 R 16968768 R 16968769 R 16968770 R 16968771 R 16968772 R 16968760 R 16968761 R 16968762 R 16968763 R 16968764 R 16968765 R 16968752 R 16968753 R 16968754 R 16968755 R 16968756 R 16968757 R 16968758 R 16968759 R 16968750 R 16968751 R parsl.providers.slurm.slurm DEBUG: Updating job 16968779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968751 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:07 proc_lsst.multi:146 [INFO] found job 16968780 in provider escience_rtx6k proc_lsst.multi INFO: found job 16968780 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968780,16968781,16968782,16968783,16968784' parsl.providers.slurm.slurm DEBUG: squeue returned 16968781 R 16968782 R 16968783 R 16968784 R 16968780 R parsl.providers.slurm.slurm DEBUG: Updating job 16968781 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968780 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:07 proc_lsst.multi:146 [INFO] found job 16968781 in provider escience_rtx6k proc_lsst.multi INFO: found job 16968781 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968780,16968781,16968782,16968783,16968784' parsl.providers.slurm.slurm DEBUG: squeue returned 16968781 R 16968782 R 16968783 R 16968784 R 16968780 R parsl.providers.slurm.slurm DEBUG: Updating job 16968781 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968780 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:08 proc_lsst.multi:146 [INFO] found job 16968782 in provider escience_rtx6k proc_lsst.multi INFO: found job 16968782 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968780,16968781,16968782,16968783,16968784' parsl.providers.slurm.slurm DEBUG: squeue returned 16968781 R 16968782 R 16968783 R 16968784 R 16968780 R parsl.providers.slurm.slurm DEBUG: Updating job 16968781 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968780 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:08 proc_lsst.multi:146 [INFO] found job 16968783 in provider escience_rtx6k proc_lsst.multi INFO: found job 16968783 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968780,16968781,16968782,16968783,16968784' parsl.providers.slurm.slurm DEBUG: squeue returned 16968781 R 16968782 R 16968783 R 16968784 R 16968780 R parsl.providers.slurm.slurm DEBUG: Updating job 16968781 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968780 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:08 proc_lsst.multi:146 [INFO] found job 16968784 in provider escience_rtx6k proc_lsst.multi INFO: found job 16968784 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968780,16968781,16968782,16968783,16968784' parsl.providers.slurm.slurm DEBUG: squeue returned 16968781 R 16968782 R 16968783 R 16968784 R 16968780 R parsl.providers.slurm.slurm DEBUG: Updating job 16968781 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968780 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:08 proc_lsst.multi:146 [INFO] found job 16968785 in provider escience_a40 proc_lsst.multi INFO: found job 16968785 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:08 proc_lsst.multi:146 [INFO] found job 16968786 in provider escience_a40 proc_lsst.multi INFO: found job 16968786 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:09 proc_lsst.multi:146 [INFO] found job 16968787 in provider escience_a40 proc_lsst.multi INFO: found job 16968787 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:09 proc_lsst.multi:146 [INFO] found job 16968788 in provider escience_a40 proc_lsst.multi INFO: found job 16968788 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:09 proc_lsst.multi:146 [INFO] found job 16968789 in provider escience_a40 proc_lsst.multi INFO: found job 16968789 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:09 proc_lsst.multi:146 [INFO] found job 16968790 in provider escience_a40 proc_lsst.multi INFO: found job 16968790 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:09 proc_lsst.multi:146 [INFO] found job 16968791 in provider escience_a40 proc_lsst.multi INFO: found job 16968791 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:10 proc_lsst.multi:146 [INFO] found job 16968792 in provider escience_a40 proc_lsst.multi INFO: found job 16968792 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:10 proc_lsst.multi:146 [INFO] found job 16968793 in provider escience_a40 proc_lsst.multi INFO: found job 16968793 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:10 proc_lsst.multi:146 [INFO] found job 16968794 in provider escience_a40 proc_lsst.multi INFO: found job 16968794 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:10 proc_lsst.multi:146 [INFO] found job 16968795 in provider escience_a40 proc_lsst.multi INFO: found job 16968795 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:10 proc_lsst.multi:146 [INFO] found job 16968796 in provider escience_a40 proc_lsst.multi INFO: found job 16968796 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:10 proc_lsst.multi:146 [INFO] found job 16968797 in provider escience_a40 proc_lsst.multi INFO: found job 16968797 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.dataflow.dflow INFO: Task 306 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ebc89f41ed1ce7350736cbc9df2be7a with result from task 306 parsl.dataflow.dflow DEBUG: Task 306 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/4923767e-2665-4ab4-b632-366b8412d0f0_isrForCrosstalkSources_855909_37.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/4923767e-2665-4ab4-b632-366b8412d0f0_isrForCrosstalkSources_855909_37.stderr parsl.dataflow.dflow INFO: Task 314 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80c000b9cc7af4bd542a2864e455e539 with result from task 314 parsl.dataflow.dflow DEBUG: Task 314 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/a76ab128-fc61-4f01-b5fb-18b2d72d3e9f_isrForCrosstalkSources_855911_41.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/a76ab128-fc61-4f01-b5fb-18b2d72d3e9f_isrForCrosstalkSources_855911_41.stderr parsl.dataflow.dflow INFO: Task 307 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c98d4315c9ba38a3dd5086375cddf23 with result from task 307 parsl.dataflow.dflow DEBUG: Task 307 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/7f1b2711-b446-42d6-9baf-dc025e0c2fdc_isrForCrosstalkSources_855821_20.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/7f1b2711-b446-42d6-9baf-dc025e0c2fdc_isrForCrosstalkSources_855821_20.stderr parsl.dataflow.dflow INFO: Task 305 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33cf69813c75aa9d075a6d94c476693e with result from task 305 parsl.dataflow.dflow DEBUG: Task 305 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/671779e8-d5a3-4e86-9d24-046deac10258_isrForCrosstalkSources_855910_41.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/671779e8-d5a3-4e86-9d24-046deac10258_isrForCrosstalkSources_855910_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:11 proc_lsst.multi:146 [INFO] found job 16968798 in provider escience_a40 proc_lsst.multi INFO: found job 16968798 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.dataflow.dflow INFO: Task 316 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e23771124066b415631e3bd860db742 with result from task 316 parsl.dataflow.dflow DEBUG: Task 316 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/b70ecc56-2c66-4a77-854c-2a633f593b06_isrForCrosstalkSources_855910_61.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/b70ecc56-2c66-4a77-854c-2a633f593b06_isrForCrosstalkSources_855910_61.stderr parsl.dataflow.dflow INFO: Task 310 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e90af6750fee9e561dd5eead46029b8a with result from task 310 parsl.dataflow.dflow DEBUG: Task 310 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/48258bbe-16cf-4006-9bc4-757f7c7f766f_isrForCrosstalkSources_855820_40.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/48258bbe-16cf-4006-9bc4-757f7c7f766f_isrForCrosstalkSources_855820_40.stderr parsl.dataflow.dflow INFO: Task 321 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2502d337e45c4bf74fedd012d639ba6e with result from task 321 parsl.dataflow.dflow DEBUG: Task 321 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/68164347-c2db-4b41-ae26-1f550dea04e6_isrForCrosstalkSources_855909_29.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/68164347-c2db-4b41-ae26-1f550dea04e6_isrForCrosstalkSources_855909_29.stderr parsl.dataflow.dflow INFO: Task 294 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee18e9dcd675362d76e039f2c0a4b5ed with result from task 294 parsl.dataflow.dflow DEBUG: Task 294 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/9c4ed7c2-2683-4dc7-b1f6-3cdf22fdf81a_isrForCrosstalkSources_855911_51.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/9c4ed7c2-2683-4dc7-b1f6-3cdf22fdf81a_isrForCrosstalkSources_855911_51.stderr parsl.dataflow.dflow INFO: Task 295 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 223f176455aa6e2ac52cbbe17f0e87e8 with result from task 295 parsl.dataflow.dflow DEBUG: Task 295 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/9c26958f-bcd6-49c4-bd32-0c7eeeaa1836_isrForCrosstalkSources_855822_23.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/9c26958f-bcd6-49c4-bd32-0c7eeeaa1836_isrForCrosstalkSources_855822_23.stderr parsl.dataflow.dflow INFO: Task 309 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 628a0d291609a70b014bfe574b1a40fa with result from task 309 parsl.dataflow.dflow DEBUG: Task 309 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/dbac9698-3296-4eba-b99b-8b621dc720a8_isrForCrosstalkSources_855822_33.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/dbac9698-3296-4eba-b99b-8b621dc720a8_isrForCrosstalkSources_855822_33.stderr parsl.dataflow.dflow INFO: Task 313 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 629b0a5a34c78c1b497515c4246cc488 with result from task 313 parsl.dataflow.dflow DEBUG: Task 313 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/2e9cd747-e5fe-41d2-8aba-c31f5e728022_isrForCrosstalkSources_855911_20.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/2e9cd747-e5fe-41d2-8aba-c31f5e728022_isrForCrosstalkSources_855911_20.stderr parsl.dataflow.dflow INFO: Task 318 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e38d51246548a31fd23f4072c1315f88 with result from task 318 parsl.dataflow.dflow DEBUG: Task 318 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/f1be4191-34fe-44f6-8caf-43bea778bf0e_isrForCrosstalkSources_855820_42.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/f1be4191-34fe-44f6-8caf-43bea778bf0e_isrForCrosstalkSources_855820_42.stderr parsl.dataflow.dflow INFO: Task 291 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e14b889ef22080f73a75bd6ae594ee5c with result from task 291 parsl.dataflow.dflow DEBUG: Task 291 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/cc77215b-6118-4b55-b537-ba7abfa1ab95_isrForCrosstalkSources_855820_33.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/cc77215b-6118-4b55-b537-ba7abfa1ab95_isrForCrosstalkSources_855820_33.stderr parsl.dataflow.dflow INFO: Task 290 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af9a515f085dc64cc5f5649041033024 with result from task 290 parsl.dataflow.dflow DEBUG: Task 290 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/1e3e312a-9bc7-49e1-9c1e-671fa2bc0746_isrForCrosstalkSources_855821_22.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/1e3e312a-9bc7-49e1-9c1e-671fa2bc0746_isrForCrosstalkSources_855821_22.stderr parsl.dataflow.dflow INFO: Task 299 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b7760d4b247a48fbf5a42bc8bf4cf00 with result from task 299 parsl.dataflow.dflow DEBUG: Task 299 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/6f27cc4c-ff62-4c4d-b5f8-e11d6ad73c0e_isrForCrosstalkSources_855822_60.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/6f27cc4c-ff62-4c4d-b5f8-e11d6ad73c0e_isrForCrosstalkSources_855822_60.stderr parsl.dataflow.dflow INFO: Task 300 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8aa82888f6ea6b537c546b58c79e1be5 with result from task 300 parsl.dataflow.dflow DEBUG: Task 300 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/e169bfe2-38b0-4407-9fdb-6e81d5747e23_isrForCrosstalkSources_855822_3.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/e169bfe2-38b0-4407-9fdb-6e81d5747e23_isrForCrosstalkSources_855822_3.stderr parsl.dataflow.dflow INFO: Task 303 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 640440fe7a408c6975b660f3ee0cb73f with result from task 303 parsl.dataflow.dflow DEBUG: Task 303 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/272e941f-5e06-451b-895f-9e10610d65bc_isrForCrosstalkSources_855822_52.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/272e941f-5e06-451b-895f-9e10610d65bc_isrForCrosstalkSources_855822_52.stderr parsl.dataflow.dflow INFO: Task 304 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b85d154e314966bde6028d21886f8b0 with result from task 304 parsl.dataflow.dflow DEBUG: Task 304 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/c5122f89-4e7a-48b3-bfeb-3330c79668d4_isrForCrosstalkSources_855910_31.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855910/c5122f89-4e7a-48b3-bfeb-3330c79668d4_isrForCrosstalkSources_855910_31.stderr parsl.dataflow.dflow INFO: Task 301 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10856ee910e83712753c46e59ad8d650 with result from task 301 parsl.dataflow.dflow DEBUG: Task 301 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/b88e8079-1f50-4e87-b00f-2e6580663231_isrForCrosstalkSources_855911_13.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/b88e8079-1f50-4e87-b00f-2e6580663231_isrForCrosstalkSources_855911_13.stderr parsl.dataflow.dflow INFO: Task 308 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98c91f1a78b282ec993a7b759f1cfc11 with result from task 308 parsl.dataflow.dflow DEBUG: Task 308 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/0fb4d167-ef8a-45a3-807f-9b6511112193_isrForCrosstalkSources_855821_46.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/0fb4d167-ef8a-45a3-807f-9b6511112193_isrForCrosstalkSources_855821_46.stderr parsl.dataflow.dflow INFO: Task 302 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13e1862f2fa12b4f759ca3efbe9441f5 with result from task 302 parsl.dataflow.dflow DEBUG: Task 302 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/4ed14335-6146-4037-9372-29ac05d84cd1_isrForCrosstalkSources_855909_42.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/4ed14335-6146-4037-9372-29ac05d84cd1_isrForCrosstalkSources_855909_42.stderr parsl.dataflow.dflow INFO: Task 311 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f73fe7263d73348044b0d00419a26af with result from task 311 parsl.dataflow.dflow DEBUG: Task 311 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/51bba081-88b9-45ce-bb37-eced9738702e_isrForCrosstalkSources_855820_60.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/51bba081-88b9-45ce-bb37-eced9738702e_isrForCrosstalkSources_855820_60.stderr parsl.dataflow.dflow INFO: Task 315 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce2cfc2a1eee061a35d1a556adee5bbd with result from task 315 parsl.dataflow.dflow DEBUG: Task 315 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/89a22a36-37a0-41f5-9d74-4b317896e716_isrForCrosstalkSources_855909_10.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/89a22a36-37a0-41f5-9d74-4b317896e716_isrForCrosstalkSources_855909_10.stderr parsl.dataflow.dflow INFO: Task 312 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7dac7bbda376d47720085b523ab551e8 with result from task 312 parsl.dataflow.dflow DEBUG: Task 312 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/1b93d2e9-64a5-42d1-9c6c-ec0fd2d90029_isrForCrosstalkSources_855822_15.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/1b93d2e9-64a5-42d1-9c6c-ec0fd2d90029_isrForCrosstalkSources_855822_15.stderr parsl.dataflow.dflow INFO: Task 319 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1701dd52fc1659046d7f77bf9dad962 with result from task 319 parsl.dataflow.dflow DEBUG: Task 319 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/21411830-0cbe-47ec-8eb8-1fe76d8d979b_isrForCrosstalkSources_855822_28.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/21411830-0cbe-47ec-8eb8-1fe76d8d979b_isrForCrosstalkSources_855822_28.stderr parsl.dataflow.dflow INFO: Task 317 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c953aa8b25a6a18fc5a8dfb08bce7cda with result from task 317 parsl.dataflow.dflow DEBUG: Task 317 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/9f562a1c-a34d-4761-822e-951dafa1e3c9_isrForCrosstalkSources_855822_58.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/9f562a1c-a34d-4761-822e-951dafa1e3c9_isrForCrosstalkSources_855822_58.stderr parsl.dataflow.dflow INFO: Task 324 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 697cb0588943e4da5e50478c2fabf012 with result from task 324 parsl.dataflow.dflow DEBUG: Task 324 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/55514120-103d-49d2-bcbc-bfcb5696b06a_isrForCrosstalkSources_855820_16.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/55514120-103d-49d2-bcbc-bfcb5696b06a_isrForCrosstalkSources_855820_16.stderr parsl.dataflow.dflow INFO: Task 296 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37fc4b160f2c3747ba7dc55caf34ec42 with result from task 296 parsl.dataflow.dflow DEBUG: Task 296 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/070a9efb-59d6-4a61-829e-8b0f80542380_isrForCrosstalkSources_855820_47.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/070a9efb-59d6-4a61-829e-8b0f80542380_isrForCrosstalkSources_855820_47.stderr parsl.dataflow.dflow INFO: Task 298 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be2315c283aa0d01faa96eaf9acaad73 with result from task 298 parsl.dataflow.dflow DEBUG: Task 298 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/947364fa-423b-4b1e-8aed-726cb44bad7c_isrForCrosstalkSources_855822_41.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/947364fa-423b-4b1e-8aed-726cb44bad7c_isrForCrosstalkSources_855822_41.stderr parsl.dataflow.dflow INFO: Task 297 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b2157342f1ec0a1f78db96715a680b83 with result from task 297 parsl.dataflow.dflow DEBUG: Task 297 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/846a9bb2-2d8e-4a2e-8fe8-5c94cdd884f9_isrForCrosstalkSources_855822_53.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/846a9bb2-2d8e-4a2e-8fe8-5c94cdd884f9_isrForCrosstalkSources_855822_53.stderr parsl.dataflow.dflow INFO: Task 293 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71f815faa129bb5a4d5fc4d3c5b7fb1f with result from task 293 parsl.dataflow.dflow DEBUG: Task 293 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/4af67321-bdc6-4bfd-a535-10c54dda467d_isrForCrosstalkSources_855909_27.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/4af67321-bdc6-4bfd-a535-10c54dda467d_isrForCrosstalkSources_855909_27.stderr parsl.dataflow.dflow INFO: Task 285 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c69664592d63539e1082df569ec66247 with result from task 285 parsl.dataflow.dflow DEBUG: Task 285 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/fd59d7e6-6502-431c-b44c-fb4fc6f07d15_isrForCrosstalkSources_855822_44.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/fd59d7e6-6502-431c-b44c-fb4fc6f07d15_isrForCrosstalkSources_855822_44.stderr parsl.dataflow.dflow INFO: Task 287 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44d44b06c4faf0e6b2ec9b8326cc0b4f with result from task 287 parsl.dataflow.dflow DEBUG: Task 287 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/911eb74c-ffee-42e4-bbcc-7c05fcaecefe_isrForCrosstalkSources_855909_50.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855909/911eb74c-ffee-42e4-bbcc-7c05fcaecefe_isrForCrosstalkSources_855909_50.stderr parsl.dataflow.dflow INFO: Task 286 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fef66d47c178ae0f6f0629b6fa83e1d2 with result from task 286 parsl.dataflow.dflow DEBUG: Task 286 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/c4251f23-8b4b-414c-84a6-e624c42d63ad_isrForCrosstalkSources_855911_49.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855911/c4251f23-8b4b-414c-84a6-e624c42d63ad_isrForCrosstalkSources_855911_49.stderr parsl.dataflow.dflow INFO: Task 292 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d48396159640a7cb5a7f6e2da8071e72 with result from task 292 parsl.dataflow.dflow DEBUG: Task 292 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/3acd1299-7784-4362-86dc-78bc156b3354_isrForCrosstalkSources_855822_46.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/3acd1299-7784-4362-86dc-78bc156b3354_isrForCrosstalkSources_855822_46.stderr parsl.dataflow.dflow INFO: Task 289 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry abd1fbfc28232298711a382ecf23fa13 with result from task 289 parsl.dataflow.dflow DEBUG: Task 289 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/43c3d0ee-a909-4279-b0d3-4c67c0955a76_isrForCrosstalkSources_855820_18.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/43c3d0ee-a909-4279-b0d3-4c67c0955a76_isrForCrosstalkSources_855820_18.stderr parsl.dataflow.dflow INFO: Task 288 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ca2582ebee230e86d32a07b64841d6a with result from task 288 parsl.dataflow.dflow DEBUG: Task 288 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/5bb84ce5-cd2a-4c62-b98d-bd775d90bb24_isrForCrosstalkSources_855820_9.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855820/5bb84ce5-cd2a-4c62-b98d-bd775d90bb24_isrForCrosstalkSources_855820_9.stderr parsl.dataflow.dflow INFO: Task 322 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83bb969def82aff3389f5f4389ea40af with result from task 322 parsl.dataflow.dflow DEBUG: Task 322 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/e26889ea-3fcd-4f14-b90d-3f438be37630_isrForCrosstalkSources_855822_31.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/e26889ea-3fcd-4f14-b90d-3f438be37630_isrForCrosstalkSources_855822_31.stderr parsl.dataflow.dflow INFO: Task 323 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0568499a4332d2367005e1b3b5ebad60 with result from task 323 parsl.dataflow.dflow DEBUG: Task 323 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/48d0536a-6458-42b3-b4bb-5af632145962_isrForCrosstalkSources_855821_50.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855821/48d0536a-6458-42b3-b4bb-5af632145962_isrForCrosstalkSources_855821_50.stderr parsl.dataflow.dflow INFO: Task 320 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23ba82224fcfaa4ef7d5febffe03b3a5 with result from task 320 parsl.dataflow.dflow INFO: DFK cleanup initiated parsl.dataflow.dflow INFO: Summary of tasks in DFK: parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0 parsl.dataflow.dflow INFO: Tasks in state States.pending: 0 parsl.dataflow.dflow INFO: Tasks in state States.running: 0 parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 372 parsl.dataflow.dflow INFO: Tasks in state States.failed: 0 parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 0 parsl.dataflow.dflow INFO: Tasks in state States.launched: 0 parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0 parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0 parsl.dataflow.dflow INFO: Tasks in state States.joining: 0 parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0 parsl.dataflow.dflow INFO: End of summary parsl.dataflow.dflow DEBUG: Task 320 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/b2d24210-8aff-4534-ae26-32851c9d0d3a_isrForCrosstalkSources_855822_51.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/submit/DEEP/20190507/A1c/science#step0/20240309T204810Z/logs/isrForCrosstalkSources/855822/b2d24210-8aff-4534-ae26-32851c9d0d3a_isrForCrosstalkSources_855822_51.stderr parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Closing job status poller parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:11 proc_lsst.multi:146 [INFO] found job 16968799 in provider escience_a40 proc_lsst.multi INFO: found job 16968799 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:11 proc_lsst.multi:146 [INFO] found job 16968800 in provider escience_a40 proc_lsst.multi INFO: found job 16968800 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:11 proc_lsst.multi:146 [INFO] found job 16968805 in provider escience_a40 proc_lsst.multi INFO: found job 16968805 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:11 proc_lsst.multi:146 [INFO] found job 16968811 in provider escience_a40 proc_lsst.multi INFO: found job 16968811 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:12 proc_lsst.multi:146 [INFO] found job 16968812 in provider escience_a40 proc_lsst.multi INFO: found job 16968812 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:12 proc_lsst.multi:146 [INFO] found job 16968813 in provider escience_a40 proc_lsst.multi INFO: found job 16968813 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968785,16968786,16968787,16968788,16968789,16968790,16968791,16968792,16968793,16968794,16968795,16968796,16968797,16968798,16968799,16968800,16968805,16968811,16968812,16968813' parsl.providers.slurm.slurm DEBUG: squeue returned 16968812 R 16968813 R 16968795 R 16968796 R 16968797 R 16968798 R 16968799 R 16968800 R 16968805 R 16968811 R 16968786 R 16968787 R 16968788 R 16968789 R 16968790 R 16968791 R 16968792 R 16968793 R 16968794 R 16968785 R parsl.providers.slurm.slurm DEBUG: Updating job 16968812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968785 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:12 proc_lsst.multi:146 [INFO] found job 16968814 in provider checkpoint proc_lsst.multi INFO: found job 16968814 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:12 proc_lsst.multi:146 [INFO] found job 16968815 in provider checkpoint proc_lsst.multi INFO: found job 16968815 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:13 proc_lsst.multi:146 [INFO] found job 16968816 in provider checkpoint proc_lsst.multi INFO: found job 16968816 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:13 proc_lsst.multi:146 [INFO] found job 16968817 in provider checkpoint proc_lsst.multi INFO: found job 16968817 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:13 proc_lsst.multi:146 [INFO] found job 16968818 in provider checkpoint proc_lsst.multi INFO: found job 16968818 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:13 proc_lsst.multi:146 [INFO] found job 16968819 in provider checkpoint proc_lsst.multi INFO: found job 16968819 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:13 proc_lsst.multi:146 [INFO] found job 16968823 in provider checkpoint proc_lsst.multi INFO: found job 16968823 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:14 proc_lsst.multi:146 [INFO] found job 16968824 in provider checkpoint proc_lsst.multi INFO: found job 16968824 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:14 proc_lsst.multi:146 [INFO] found job 16968825 in provider checkpoint proc_lsst.multi INFO: found job 16968825 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:14 proc_lsst.multi:146 [INFO] found job 16968826 in provider checkpoint proc_lsst.multi INFO: found job 16968826 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:14 proc_lsst.multi:146 [INFO] found job 16968827 in provider checkpoint proc_lsst.multi INFO: found job 16968827 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:14 proc_lsst.multi:146 [INFO] found job 16968828 in provider checkpoint proc_lsst.multi INFO: found job 16968828 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:15 proc_lsst.multi:146 [INFO] found job 16968829 in provider checkpoint proc_lsst.multi INFO: found job 16968829 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:15 proc_lsst.multi:146 [INFO] found job 16968830 in provider checkpoint proc_lsst.multi INFO: found job 16968830 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:15 proc_lsst.multi:146 [INFO] found job 16968831 in provider checkpoint proc_lsst.multi INFO: found job 16968831 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:15 proc_lsst.multi:146 [INFO] found job 16968832 in provider checkpoint proc_lsst.multi INFO: found job 16968832 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:15 proc_lsst.multi:146 [INFO] found job 16968833 in provider checkpoint proc_lsst.multi INFO: found job 16968833 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:16 proc_lsst.multi:146 [INFO] found job 16968834 in provider checkpoint proc_lsst.multi INFO: found job 16968834 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:16 proc_lsst.multi:146 [INFO] found job 16968835 in provider checkpoint proc_lsst.multi INFO: found job 16968835 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:16 proc_lsst.multi:146 [INFO] found job 16968836 in provider checkpoint proc_lsst.multi INFO: found job 16968836 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:16 proc_lsst.multi:146 [INFO] found job 16968837 in provider checkpoint proc_lsst.multi INFO: found job 16968837 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:16 proc_lsst.multi:146 [INFO] found job 16968838 in provider checkpoint proc_lsst.multi INFO: found job 16968838 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:17 proc_lsst.multi:146 [INFO] found job 16968839 in provider checkpoint proc_lsst.multi INFO: found job 16968839 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:17 proc_lsst.multi:146 [INFO] found job 16968840 in provider checkpoint proc_lsst.multi INFO: found job 16968840 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:17 proc_lsst.multi:146 [INFO] found job 16968841 in provider checkpoint proc_lsst.multi INFO: found job 16968841 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:17 proc_lsst.multi:146 [INFO] found job 16968842 in provider checkpoint proc_lsst.multi INFO: found job 16968842 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:17 proc_lsst.multi:146 [INFO] found job 16968843 in provider checkpoint proc_lsst.multi INFO: found job 16968843 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:18 proc_lsst.multi:146 [INFO] found job 16968844 in provider checkpoint proc_lsst.multi INFO: found job 16968844 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:18 proc_lsst.multi:146 [INFO] found job 16968845 in provider checkpoint proc_lsst.multi INFO: found job 16968845 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:18 proc_lsst.multi:146 [INFO] found job 16968846 in provider checkpoint proc_lsst.multi INFO: found job 16968846 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:18 proc_lsst.multi:146 [INFO] found job 16968847 in provider checkpoint proc_lsst.multi INFO: found job 16968847 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:19 proc_lsst.multi:146 [INFO] found job 16968848 in provider checkpoint proc_lsst.multi INFO: found job 16968848 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:19 proc_lsst.multi:146 [INFO] found job 16968849 in provider checkpoint proc_lsst.multi INFO: found job 16968849 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:19 proc_lsst.multi:146 [INFO] found job 16968850 in provider checkpoint proc_lsst.multi INFO: found job 16968850 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:19 proc_lsst.multi:146 [INFO] found job 16968851 in provider checkpoint proc_lsst.multi INFO: found job 16968851 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:19 proc_lsst.multi:146 [INFO] found job 16968852 in provider checkpoint proc_lsst.multi INFO: found job 16968852 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING 2024-03-09 20:52:20 proc_lsst.multi:146 [INFO] found job 16968853 in provider checkpoint proc_lsst.multi INFO: found job 16968853 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16968814,16968815,16968816,16968817,16968818,16968819,16968823,16968824,16968825,16968826,16968827,16968828,16968829,16968830,16968831,16968832,16968833,16968834,16968835,16968836,16968837,16968838,16968839,16968840,16968841,16968842,16968843,16968844,16968845,16968846,16968847,16968848,16968849,16968850,16968851,16968852,16968853' parsl.providers.slurm.slurm DEBUG: squeue returned 16968814 R 16968815 R 16968816 R 16968817 R 16968818 R 16968819 R 16968823 R 16968824 R 16968825 R 16968826 R 16968827 R 16968828 R 16968829 R 16968830 R 16968831 R 16968832 R 16968833 R 16968834 R 16968835 R 16968836 R 16968837 R 16968838 R 16968839 R 16968840 R 16968841 R 16968842 R 16968843 R 16968844 R 16968845 R 16968846 R 16968847 R 16968848 R 16968849 R 16968850 R 16968851 R 16968852 R 16968853 R parsl.providers.slurm.slurm DEBUG: Updating job 16968814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16968853 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 0 parsl.dataflow.strategy DEBUG: Executor multi has 0 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 1: Executor has no active tasks parsl.dataflow.strategy DEBUG: Strategy case 1b: Executor has no active tasks, and more (93) than minimum blocks (0) parsl.dataflow.strategy DEBUG: Starting idle timer for executor. If idle time exceeds 120.0s, blocks will be scaled in parsl.dataflow.strategy DEBUG: Idle time 1.6689300537109375e-06s is less than max_idletime 120.0s for executor multi; not scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22762785682704 parsl.dataflow.dflow INFO: Terminated job status poller parsl.dataflow.dflow INFO: Scaling in and shutting down executors parsl.dataflow.dflow INFO: Scaling in executor multi parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=93, block_ids=[] parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 93 blocks parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b31fc8c26217 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b31fc8c26217 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 101e9fc4fe60 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 101e9fc4fe60 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e181216e0989 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e181216e0989 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fd83571ed925 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fd83571ed925 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 494f07095435 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 494f07095435 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e550544801bf parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e550544801bf parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2d7362772050 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2d7362772050 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cbcb26fa8f92 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cbcb26fa8f92 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 41c760cd2793 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 41c760cd2793 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0df4b789bdbe parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0df4b789bdbe parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0bb3da40ee79 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0bb3da40ee79 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 45101c18b6cc parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 45101c18b6cc parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 37bb5600263e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 37bb5600263e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ebb7bced0018 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ebb7bced0018 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e44b7dc5ab19 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e44b7dc5ab19 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 175926bdb772 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 175926bdb772 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 807eb53da6bd parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 807eb53da6bd parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d788fae8b8e9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d788fae8b8e9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bf77b8e4f3ae parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bf77b8e4f3ae parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 70a80632104d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 70a80632104d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 46eb7f6bf942 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 46eb7f6bf942 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 84e38445d546 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 84e38445d546 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a1b27e0259bb parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a1b27e0259bb parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 53670d23865d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 53670d23865d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e7466ebde48b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e7466ebde48b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2339993ec06f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2339993ec06f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 87c4b38edcc6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 87c4b38edcc6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 997b74c2a480 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 997b74c2a480 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ce5fbb41a901 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ce5fbb41a901 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 07cd6d7f2a5d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 07cd6d7f2a5d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c220588104b7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c220588104b7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2b7a6eba3cbc parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2b7a6eba3cbc parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f4829aceb838 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f4829aceb838 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 708cdadc283e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 708cdadc283e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8f5c2af04d20 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8f5c2af04d20 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d0ea7976d0a2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d0ea7976d0a2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3865ac05de02 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3865ac05de02 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 50843723c1fc parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 50843723c1fc parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6192fe001cb6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6192fe001cb6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: efd6721ac4a1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: efd6721ac4a1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 107e73016158 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 107e73016158 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f91945335342 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f91945335342 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fe0bf2ca5733 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fe0bf2ca5733 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: abcb2928bec4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: abcb2928bec4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 426ca6dba557 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 426ca6dba557 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d43dc02080dc parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d43dc02080dc parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ab92ba4b51e0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ab92ba4b51e0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5e850063cb43 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5e850063cb43 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7b9140c45e2b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7b9140c45e2b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5a215735ab47 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5a215735ab47 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 671f8e8be6a5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 671f8e8be6a5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a9bf26706f0f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a9bf26706f0f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e55769dfeb6a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e55769dfeb6a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0f97bf1650d8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0f97bf1650d8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c6036066a098 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c6036066a098 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3b3548c4a25c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3b3548c4a25c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a76cc1dd6f21 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a76cc1dd6f21 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c908c961ec79 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c908c961ec79 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a92f6be42af7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a92f6be42af7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c65e481cbf57 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c65e481cbf57 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5d0ff1757d80 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5d0ff1757d80 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9d9c1d1a60db parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9d9c1d1a60db parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1fdf3ae56d57 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1fdf3ae56d57 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5e532953ad47 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5e532953ad47 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9f4d8f896bdc parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9f4d8f896bdc parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6fafe1127880 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6fafe1127880 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ec177fd6f6a9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ec177fd6f6a9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7f5603a00fa6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7f5603a00fa6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 035748b1262c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 035748b1262c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a5800c74b6ad parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a5800c74b6ad parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3ffba722f294 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3ffba722f294 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2e7bcefd07aa parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2e7bcefd07aa parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e26e44667b6b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e26e44667b6b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 597b4d515f47 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 597b4d515f47 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 35bf94400f9d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 35bf94400f9d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f7d116ec0a25 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f7d116ec0a25 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bf4e13c3895e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bf4e13c3895e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5d76abde8c2e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5d76abde8c2e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9d849b4e0852 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9d849b4e0852 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 131f163464e4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 131f163464e4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2ab15100ddc7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2ab15100ddc7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4588627ae321 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4588627ae321 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 004e4acd4190 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 004e4acd4190 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0074b70044c3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0074b70044c3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0bdd5c5eed53 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0bdd5c5eed53 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5fd77f424cb4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5fd77f424cb4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 783e2f6be006 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 783e2f6be006 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 41459ae0476f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 41459ae0476f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a9f1f40a4158 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a9f1f40a4158 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3405ae8d9181 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3405ae8d9181 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0a31321aa071 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0a31321aa071 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a314d7773017 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a314d7773017 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6d78362a7d28 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6d78362a7d28 2024-03-09 20:52:20 proc_lsst.multi:146 [INFO] found job 16968838 in provider checkpoint proc_lsst.multi INFO: found job 16968838 in provider checkpoint 2024-03-09 20:52:20 proc_lsst.multi:201 [INFO] cancelling 16968838 on provider checkpoint proc_lsst.multi INFO: cancelling 16968838 on provider checkpoint 2024-03-09 20:52:20 proc_lsst.multi:146 [INFO] found job 16968837 in provider checkpoint proc_lsst.multi INFO: found job 16968837 in provider checkpoint 2024-03-09 20:52:20 proc_lsst.multi:201 [INFO] cancelling 16968837 on provider checkpoint proc_lsst.multi INFO: cancelling 16968837 on provider checkpoint 2024-03-09 20:52:20 proc_lsst.multi:146 [INFO] found job 16968836 in provider checkpoint proc_lsst.multi INFO: found job 16968836 in provider checkpoint 2024-03-09 20:52:20 proc_lsst.multi:201 [INFO] cancelling 16968836 on provider checkpoint proc_lsst.multi INFO: cancelling 16968836 on provider checkpoint 2024-03-09 20:52:20 proc_lsst.multi:146 [INFO] found job 16968839 in provider checkpoint proc_lsst.multi INFO: found job 16968839 in provider checkpoint 2024-03-09 20:52:20 proc_lsst.multi:201 [INFO] cancelling 16968839 on provider checkpoint proc_lsst.multi INFO: cancelling 16968839 on provider checkpoint 2024-03-09 20:52:21 proc_lsst.multi:146 [INFO] found job 16968840 in provider checkpoint proc_lsst.multi INFO: found job 16968840 in provider checkpoint 2024-03-09 20:52:21 proc_lsst.multi:201 [INFO] cancelling 16968840 on provider checkpoint proc_lsst.multi INFO: cancelling 16968840 on provider checkpoint 2024-03-09 20:52:21 proc_lsst.multi:146 [INFO] found job 16968834 in provider checkpoint proc_lsst.multi INFO: found job 16968834 in provider checkpoint 2024-03-09 20:52:21 proc_lsst.multi:201 [INFO] cancelling 16968834 on provider checkpoint proc_lsst.multi INFO: cancelling 16968834 on provider checkpoint 2024-03-09 20:52:21 proc_lsst.multi:146 [INFO] found job 16968835 in provider checkpoint proc_lsst.multi INFO: found job 16968835 in provider checkpoint 2024-03-09 20:52:21 proc_lsst.multi:201 [INFO] cancelling 16968835 on provider checkpoint proc_lsst.multi INFO: cancelling 16968835 on provider checkpoint 2024-03-09 20:52:21 proc_lsst.multi:146 [INFO] found job 16968785 in provider escience_a40 proc_lsst.multi INFO: found job 16968785 in provider escience_a40 2024-03-09 20:52:21 proc_lsst.multi:201 [INFO] cancelling 16968785 on provider escience_a40 proc_lsst.multi INFO: cancelling 16968785 on provider escience_a40 2024-03-09 20:52:21 proc_lsst.multi:146 [INFO] found job 16968831 in provider checkpoint proc_lsst.multi INFO: found job 16968831 in provider checkpoint 2024-03-09 20:52:21 proc_lsst.multi:201 [INFO] cancelling 16968831 on provider checkpoint proc_lsst.multi INFO: cancelling 16968831 on provider checkpoint 2024-03-09 20:52:21 proc_lsst.multi:146 [INFO] found job 16968819 in provider checkpoint proc_lsst.multi INFO: found job 16968819 in provider checkpoint 2024-03-09 20:52:21 proc_lsst.multi:201 [INFO] cancelling 16968819 on provider checkpoint proc_lsst.multi INFO: cancelling 16968819 on provider checkpoint 2024-03-09 20:52:22 proc_lsst.multi:146 [INFO] found job 16968823 in provider checkpoint proc_lsst.multi INFO: found job 16968823 in provider checkpoint 2024-03-09 20:52:22 proc_lsst.multi:201 [INFO] cancelling 16968823 on provider checkpoint proc_lsst.multi INFO: cancelling 16968823 on provider checkpoint 2024-03-09 20:52:22 proc_lsst.multi:146 [INFO] found job 16968824 in provider checkpoint proc_lsst.multi INFO: found job 16968824 in provider checkpoint 2024-03-09 20:52:22 proc_lsst.multi:201 [INFO] cancelling 16968824 on provider checkpoint proc_lsst.multi INFO: cancelling 16968824 on provider checkpoint 2024-03-09 20:52:22 proc_lsst.multi:146 [INFO] found job 16968825 in provider checkpoint proc_lsst.multi INFO: found job 16968825 in provider checkpoint 2024-03-09 20:52:22 proc_lsst.multi:201 [INFO] cancelling 16968825 on provider checkpoint proc_lsst.multi INFO: cancelling 16968825 on provider checkpoint 2024-03-09 20:52:22 proc_lsst.multi:146 [INFO] found job 16968817 in provider checkpoint proc_lsst.multi INFO: found job 16968817 in provider checkpoint 2024-03-09 20:52:22 proc_lsst.multi:201 [INFO] cancelling 16968817 on provider checkpoint proc_lsst.multi INFO: cancelling 16968817 on provider checkpoint 2024-03-09 20:52:22 proc_lsst.multi:146 [INFO] found job 16968818 in provider checkpoint proc_lsst.multi INFO: found job 16968818 in provider checkpoint 2024-03-09 20:52:22 proc_lsst.multi:201 [INFO] cancelling 16968818 on provider checkpoint proc_lsst.multi INFO: cancelling 16968818 on provider checkpoint 2024-03-09 20:52:22 proc_lsst.multi:146 [INFO] found job 16968815 in provider checkpoint proc_lsst.multi INFO: found job 16968815 in provider checkpoint 2024-03-09 20:52:22 proc_lsst.multi:201 [INFO] cancelling 16968815 on provider checkpoint proc_lsst.multi INFO: cancelling 16968815 on provider checkpoint 2024-03-09 20:52:23 proc_lsst.multi:146 [INFO] found job 16968816 in provider checkpoint proc_lsst.multi INFO: found job 16968816 in provider checkpoint 2024-03-09 20:52:23 proc_lsst.multi:201 [INFO] cancelling 16968816 on provider checkpoint proc_lsst.multi INFO: cancelling 16968816 on provider checkpoint 2024-03-09 20:52:23 proc_lsst.multi:146 [INFO] found job 16968827 in provider checkpoint proc_lsst.multi INFO: found job 16968827 in provider checkpoint 2024-03-09 20:52:23 proc_lsst.multi:201 [INFO] cancelling 16968827 on provider checkpoint proc_lsst.multi INFO: cancelling 16968827 on provider checkpoint 2024-03-09 20:52:23 proc_lsst.multi:146 [INFO] found job 16968814 in provider checkpoint proc_lsst.multi INFO: found job 16968814 in provider checkpoint 2024-03-09 20:52:23 proc_lsst.multi:201 [INFO] cancelling 16968814 on provider checkpoint proc_lsst.multi INFO: cancelling 16968814 on provider checkpoint 2024-03-09 20:52:23 proc_lsst.multi:146 [INFO] found job 16968842 in provider checkpoint proc_lsst.multi INFO: found job 16968842 in provider checkpoint 2024-03-09 20:52:23 proc_lsst.multi:201 [INFO] cancelling 16968842 on provider checkpoint proc_lsst.multi INFO: cancelling 16968842 on provider checkpoint 2024-03-09 20:52:23 proc_lsst.multi:146 [INFO] found job 16968843 in provider checkpoint proc_lsst.multi INFO: found job 16968843 in provider checkpoint 2024-03-09 20:52:23 proc_lsst.multi:201 [INFO] cancelling 16968843 on provider checkpoint proc_lsst.multi INFO: cancelling 16968843 on provider checkpoint 2024-03-09 20:52:23 proc_lsst.multi:146 [INFO] found job 16968848 in provider checkpoint proc_lsst.multi INFO: found job 16968848 in provider checkpoint 2024-03-09 20:52:23 proc_lsst.multi:201 [INFO] cancelling 16968848 on provider checkpoint proc_lsst.multi INFO: cancelling 16968848 on provider checkpoint 2024-03-09 20:52:24 proc_lsst.multi:146 [INFO] found job 16968847 in provider checkpoint proc_lsst.multi INFO: found job 16968847 in provider checkpoint 2024-03-09 20:52:24 proc_lsst.multi:201 [INFO] cancelling 16968847 on provider checkpoint proc_lsst.multi INFO: cancelling 16968847 on provider checkpoint 2024-03-09 20:52:24 proc_lsst.multi:146 [INFO] found job 16968841 in provider checkpoint proc_lsst.multi INFO: found job 16968841 in provider checkpoint 2024-03-09 20:52:24 proc_lsst.multi:201 [INFO] cancelling 16968841 on provider checkpoint proc_lsst.multi INFO: cancelling 16968841 on provider checkpoint 2024-03-09 20:52:24 proc_lsst.multi:146 [INFO] found job 16968850 in provider checkpoint proc_lsst.multi INFO: found job 16968850 in provider checkpoint 2024-03-09 20:52:24 proc_lsst.multi:201 [INFO] cancelling 16968850 on provider checkpoint proc_lsst.multi INFO: cancelling 16968850 on provider checkpoint 2024-03-09 20:52:24 proc_lsst.multi:146 [INFO] found job 16968852 in provider checkpoint proc_lsst.multi INFO: found job 16968852 in provider checkpoint 2024-03-09 20:52:24 proc_lsst.multi:201 [INFO] cancelling 16968852 on provider checkpoint proc_lsst.multi INFO: cancelling 16968852 on provider checkpoint 2024-03-09 20:52:24 proc_lsst.multi:146 [INFO] found job 16968851 in provider checkpoint proc_lsst.multi INFO: found job 16968851 in provider checkpoint 2024-03-09 20:52:24 proc_lsst.multi:201 [INFO] cancelling 16968851 on provider checkpoint proc_lsst.multi INFO: cancelling 16968851 on provider checkpoint 2024-03-09 20:52:25 proc_lsst.multi:146 [INFO] found job 16968849 in provider checkpoint proc_lsst.multi INFO: found job 16968849 in provider checkpoint 2024-03-09 20:52:25 proc_lsst.multi:201 [INFO] cancelling 16968849 on provider checkpoint proc_lsst.multi INFO: cancelling 16968849 on provider checkpoint 2024-03-09 20:52:25 proc_lsst.multi:146 [INFO] found job 16968853 in provider checkpoint proc_lsst.multi INFO: found job 16968853 in provider checkpoint 2024-03-09 20:52:25 proc_lsst.multi:201 [INFO] cancelling 16968853 on provider checkpoint proc_lsst.multi INFO: cancelling 16968853 on provider checkpoint 2024-03-09 20:52:25 proc_lsst.multi:146 [INFO] found job 16968845 in provider checkpoint proc_lsst.multi INFO: found job 16968845 in provider checkpoint 2024-03-09 20:52:25 proc_lsst.multi:201 [INFO] cancelling 16968845 on provider checkpoint proc_lsst.multi INFO: cancelling 16968845 on provider checkpoint 2024-03-09 20:52:25 proc_lsst.multi:146 [INFO] found job 16968846 in provider checkpoint proc_lsst.multi INFO: found job 16968846 in provider checkpoint 2024-03-09 20:52:25 proc_lsst.multi:201 [INFO] cancelling 16968846 on provider checkpoint proc_lsst.multi INFO: cancelling 16968846 on provider checkpoint 2024-03-09 20:52:25 proc_lsst.multi:146 [INFO] found job 16968794 in provider escience_a40 proc_lsst.multi INFO: found job 16968794 in provider escience_a40 2024-03-09 20:52:25 proc_lsst.multi:201 [INFO] cancelling 16968794 on provider escience_a40 proc_lsst.multi INFO: cancelling 16968794 on provider escience_a40 2024-03-09 20:52:25 proc_lsst.multi:146 [INFO] found job 16968791 in provider escience_a40 proc_lsst.multi INFO: found job 16968791 in provider escience_a40 2024-03-09 20:52:25 proc_lsst.multi:201 [INFO] cancelling 16968791 on provider escience_a40 proc_lsst.multi INFO: cancelling 16968791 on provider escience_a40 2024-03-09 20:52:26 proc_lsst.multi:146 [INFO] found job 16968786 in provider escience_a40 proc_lsst.multi INFO: found job 16968786 in provider escience_a40 2024-03-09 20:52:26 proc_lsst.multi:201 [INFO] cancelling 16968786 on provider escience_a40 proc_lsst.multi INFO: cancelling 16968786 on provider escience_a40 2024-03-09 20:52:26 proc_lsst.multi:146 [INFO] found job 16968793 in provider escience_a40 proc_lsst.multi INFO: found job 16968793 in provider escience_a40 2024-03-09 20:52:26 proc_lsst.multi:201 [INFO] cancelling 16968793 on provider escience_a40 proc_lsst.multi INFO: cancelling 16968793 on provider escience_a40 2024-03-09 20:52:26 proc_lsst.multi:146 [INFO] found job 16968795 in provider escience_a40 proc_lsst.multi INFO: found job 16968795 in provider escience_a40 2024-03-09 20:52:26 proc_lsst.multi:201 [INFO] cancelling 16968795 on provider escience_a40 proc_lsst.multi INFO: cancelling 16968795 on provider escience_a40 2024-03-09 20:52:26 proc_lsst.multi:146 [INFO] found job 16968759 in provider astro proc_lsst.multi INFO: found job 16968759 in provider astro 2024-03-09 20:52:26 proc_lsst.multi:201 [INFO] cancelling 16968759 on provider astro proc_lsst.multi INFO: cancelling 16968759 on provider astro 2024-03-09 20:52:26 proc_lsst.multi:146 [INFO] found job 16968792 in provider escience_a40 proc_lsst.multi INFO: found job 16968792 in provider escience_a40 2024-03-09 20:52:26 proc_lsst.multi:201 [INFO] cancelling 16968792 on provider escience_a40 proc_lsst.multi INFO: cancelling 16968792 on provider escience_a40 2024-03-09 20:52:26 proc_lsst.multi:146 [INFO] found job 16968790 in provider escience_a40 proc_lsst.multi INFO: found job 16968790 in provider escience_a40 2024-03-09 20:52:26 proc_lsst.multi:201 [INFO] cancelling 16968790 on provider escience_a40 proc_lsst.multi INFO: cancelling 16968790 on provider escience_a40 2024-03-09 20:52:27 proc_lsst.multi:146 [INFO] found job 16968788 in provider escience_a40 proc_lsst.multi INFO: found job 16968788 in provider escience_a40 2024-03-09 20:52:27 proc_lsst.multi:201 [INFO] cancelling 16968788 on provider escience_a40 proc_lsst.multi INFO: cancelling 16968788 on provider escience_a40 2024-03-09 20:52:27 proc_lsst.multi:146 [INFO] found job 16968761 in provider astro proc_lsst.multi INFO: found job 16968761 in provider astro 2024-03-09 20:52:27 proc_lsst.multi:201 [INFO] cancelling 16968761 on provider astro proc_lsst.multi INFO: cancelling 16968761 on provider astro 2024-03-09 20:52:27 proc_lsst.multi:146 [INFO] found job 16968754 in provider astro proc_lsst.multi INFO: found job 16968754 in provider astro 2024-03-09 20:52:27 proc_lsst.multi:201 [INFO] cancelling 16968754 on provider astro proc_lsst.multi INFO: cancelling 16968754 on provider astro 2024-03-09 20:52:27 proc_lsst.multi:146 [INFO] found job 16968756 in provider astro proc_lsst.multi INFO: found job 16968756 in provider astro 2024-03-09 20:52:27 proc_lsst.multi:201 [INFO] cancelling 16968756 on provider astro proc_lsst.multi INFO: cancelling 16968756 on provider astro 2024-03-09 20:52:27 proc_lsst.multi:146 [INFO] found job 16968758 in provider astro proc_lsst.multi INFO: found job 16968758 in provider astro 2024-03-09 20:52:27 proc_lsst.multi:201 [INFO] cancelling 16968758 on provider astro proc_lsst.multi INFO: cancelling 16968758 on provider astro 2024-03-09 20:52:28 proc_lsst.multi:146 [INFO] found job 16968772 in provider astro proc_lsst.multi INFO: found job 16968772 in provider astro 2024-03-09 20:52:28 proc_lsst.multi:201 [INFO] cancelling 16968772 on provider astro proc_lsst.multi INFO: cancelling 16968772 on provider astro 2024-03-09 20:52:28 proc_lsst.multi:146 [INFO] found job 16968770 in provider astro proc_lsst.multi INFO: found job 16968770 in provider astro 2024-03-09 20:52:28 proc_lsst.multi:201 [INFO] cancelling 16968770 on provider astro proc_lsst.multi INFO: cancelling 16968770 on provider astro 2024-03-09 20:52:28 proc_lsst.multi:146 [INFO] found job 16968789 in provider escience_a40 proc_lsst.multi INFO: found job 16968789 in provider escience_a40 2024-03-09 20:52:28 proc_lsst.multi:201 [INFO] cancelling 16968789 on provider escience_a40 proc_lsst.multi INFO: cancelling 16968789 on provider escience_a40 2024-03-09 20:52:28 proc_lsst.multi:146 [INFO] found job 16968844 in provider checkpoint proc_lsst.multi INFO: found job 16968844 in provider checkpoint 2024-03-09 20:52:28 proc_lsst.multi:201 [INFO] cancelling 16968844 on provider checkpoint proc_lsst.multi INFO: cancelling 16968844 on provider checkpoint 2024-03-09 20:52:28 proc_lsst.multi:146 [INFO] found job 16968777 in provider astro proc_lsst.multi INFO: found job 16968777 in provider astro 2024-03-09 20:52:28 proc_lsst.multi:201 [INFO] cancelling 16968777 on provider astro proc_lsst.multi INFO: cancelling 16968777 on provider astro 2024-03-09 20:52:28 proc_lsst.multi:146 [INFO] found job 16968773 in provider astro proc_lsst.multi INFO: found job 16968773 in provider astro 2024-03-09 20:52:28 proc_lsst.multi:201 [INFO] cancelling 16968773 on provider astro proc_lsst.multi INFO: cancelling 16968773 on provider astro 2024-03-09 20:52:29 proc_lsst.multi:146 [INFO] found job 16968775 in provider astro proc_lsst.multi INFO: found job 16968775 in provider astro 2024-03-09 20:52:29 proc_lsst.multi:201 [INFO] cancelling 16968775 on provider astro proc_lsst.multi INFO: cancelling 16968775 on provider astro 2024-03-09 20:52:29 proc_lsst.multi:146 [INFO] found job 16968760 in provider astro proc_lsst.multi INFO: found job 16968760 in provider astro 2024-03-09 20:52:29 proc_lsst.multi:201 [INFO] cancelling 16968760 on provider astro proc_lsst.multi INFO: cancelling 16968760 on provider astro 2024-03-09 20:52:29 proc_lsst.multi:146 [INFO] found job 16968764 in provider astro proc_lsst.multi INFO: found job 16968764 in provider astro 2024-03-09 20:52:29 proc_lsst.multi:201 [INFO] cancelling 16968764 on provider astro proc_lsst.multi INFO: cancelling 16968764 on provider astro 2024-03-09 20:52:29 proc_lsst.multi:146 [INFO] found job 16968771 in provider astro proc_lsst.multi INFO: found job 16968771 in provider astro 2024-03-09 20:52:29 proc_lsst.multi:201 [INFO] cancelling 16968771 on provider astro proc_lsst.multi INFO: cancelling 16968771 on provider astro 2024-03-09 20:52:29 proc_lsst.multi:146 [INFO] found job 16968755 in provider astro proc_lsst.multi INFO: found job 16968755 in provider astro 2024-03-09 20:52:29 proc_lsst.multi:201 [INFO] cancelling 16968755 on provider astro proc_lsst.multi INFO: cancelling 16968755 on provider astro 2024-03-09 20:52:30 proc_lsst.multi:146 [INFO] found job 35332 in provider local proc_lsst.multi INFO: found job 35332 in provider local 2024-03-09 20:52:30 proc_lsst.multi:201 [INFO] cancelling 35332 on provider local proc_lsst.multi INFO: cancelling 35332 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 35332 2024-03-09 20:52:30 proc_lsst.multi:146 [INFO] found job 16968767 in provider astro proc_lsst.multi INFO: found job 16968767 in provider astro 2024-03-09 20:52:30 proc_lsst.multi:201 [INFO] cancelling 16968767 on provider astro proc_lsst.multi INFO: cancelling 16968767 on provider astro 2024-03-09 20:52:30 proc_lsst.multi:146 [INFO] found job 16968796 in provider escience_a40 proc_lsst.multi INFO: found job 16968796 in provider escience_a40 2024-03-09 20:52:30 proc_lsst.multi:201 [INFO] cancelling 16968796 on provider escience_a40 proc_lsst.multi INFO: cancelling 16968796 on provider escience_a40 2024-03-09 20:52:30 proc_lsst.multi:146 [INFO] found job 16968805 in provider escience_a40 proc_lsst.multi INFO: found job 16968805 in provider escience_a40 2024-03-09 20:52:30 proc_lsst.multi:201 [INFO] cancelling 16968805 on provider escience_a40 proc_lsst.multi INFO: cancelling 16968805 on provider escience_a40 2024-03-09 20:52:30 proc_lsst.multi:146 [INFO] found job 16968776 in provider astro proc_lsst.multi INFO: found job 16968776 in provider astro 2024-03-09 20:52:30 proc_lsst.multi:201 [INFO] cancelling 16968776 on provider astro proc_lsst.multi INFO: cancelling 16968776 on provider astro 2024-03-09 20:52:30 proc_lsst.multi:146 [INFO] found job 16968787 in provider escience_a40 proc_lsst.multi INFO: found job 16968787 in provider escience_a40 2024-03-09 20:52:30 proc_lsst.multi:201 [INFO] cancelling 16968787 on provider escience_a40 proc_lsst.multi INFO: cancelling 16968787 on provider escience_a40 2024-03-09 20:52:31 proc_lsst.multi:146 [INFO] found job 16968811 in provider escience_a40 proc_lsst.multi INFO: found job 16968811 in provider escience_a40 2024-03-09 20:52:31 proc_lsst.multi:201 [INFO] cancelling 16968811 on provider escience_a40 proc_lsst.multi INFO: cancelling 16968811 on provider escience_a40 2024-03-09 20:52:31 proc_lsst.multi:146 [INFO] found job 16968800 in provider escience_a40 proc_lsst.multi INFO: found job 16968800 in provider escience_a40 2024-03-09 20:52:31 proc_lsst.multi:201 [INFO] cancelling 16968800 on provider escience_a40 proc_lsst.multi INFO: cancelling 16968800 on provider escience_a40 2024-03-09 20:52:31 proc_lsst.multi:146 [INFO] found job 16968779 in provider astro proc_lsst.multi INFO: found job 16968779 in provider astro 2024-03-09 20:52:31 proc_lsst.multi:201 [INFO] cancelling 16968779 on provider astro proc_lsst.multi INFO: cancelling 16968779 on provider astro 2024-03-09 20:52:31 proc_lsst.multi:146 [INFO] found job 16968768 in provider astro proc_lsst.multi INFO: found job 16968768 in provider astro 2024-03-09 20:52:31 proc_lsst.multi:201 [INFO] cancelling 16968768 on provider astro proc_lsst.multi INFO: cancelling 16968768 on provider astro 2024-03-09 20:52:31 proc_lsst.multi:146 [INFO] found job 16968763 in provider astro proc_lsst.multi INFO: found job 16968763 in provider astro 2024-03-09 20:52:31 proc_lsst.multi:201 [INFO] cancelling 16968763 on provider astro proc_lsst.multi INFO: cancelling 16968763 on provider astro 2024-03-09 20:52:32 proc_lsst.multi:146 [INFO] found job 16968762 in provider astro proc_lsst.multi INFO: found job 16968762 in provider astro 2024-03-09 20:52:32 proc_lsst.multi:201 [INFO] cancelling 16968762 on provider astro proc_lsst.multi INFO: cancelling 16968762 on provider astro 2024-03-09 20:52:32 proc_lsst.multi:146 [INFO] found job 16968774 in provider astro proc_lsst.multi INFO: found job 16968774 in provider astro 2024-03-09 20:52:32 proc_lsst.multi:201 [INFO] cancelling 16968774 on provider astro proc_lsst.multi INFO: cancelling 16968774 on provider astro 2024-03-09 20:52:32 proc_lsst.multi:146 [INFO] found job 16968766 in provider astro proc_lsst.multi INFO: found job 16968766 in provider astro 2024-03-09 20:52:32 proc_lsst.multi:201 [INFO] cancelling 16968766 on provider astro proc_lsst.multi INFO: cancelling 16968766 on provider astro 2024-03-09 20:52:32 proc_lsst.multi:146 [INFO] found job 16968778 in provider astro proc_lsst.multi INFO: found job 16968778 in provider astro 2024-03-09 20:52:32 proc_lsst.multi:201 [INFO] cancelling 16968778 on provider astro proc_lsst.multi INFO: cancelling 16968778 on provider astro 2024-03-09 20:52:32 proc_lsst.multi:146 [INFO] found job 16968797 in provider escience_a40 proc_lsst.multi INFO: found job 16968797 in provider escience_a40 2024-03-09 20:52:32 proc_lsst.multi:201 [INFO] cancelling 16968797 on provider escience_a40 proc_lsst.multi INFO: cancelling 16968797 on provider escience_a40 2024-03-09 20:52:32 proc_lsst.multi:146 [INFO] found job 16968812 in provider escience_a40 proc_lsst.multi INFO: found job 16968812 in provider escience_a40 2024-03-09 20:52:32 proc_lsst.multi:201 [INFO] cancelling 16968812 on provider escience_a40 proc_lsst.multi INFO: cancelling 16968812 on provider escience_a40 2024-03-09 20:52:33 proc_lsst.multi:146 [INFO] found job 16968769 in provider astro proc_lsst.multi INFO: found job 16968769 in provider astro 2024-03-09 20:52:33 proc_lsst.multi:201 [INFO] cancelling 16968769 on provider astro proc_lsst.multi INFO: cancelling 16968769 on provider astro 2024-03-09 20:52:33 proc_lsst.multi:146 [INFO] found job 16968765 in provider astro proc_lsst.multi INFO: found job 16968765 in provider astro 2024-03-09 20:52:33 proc_lsst.multi:201 [INFO] cancelling 16968765 on provider astro proc_lsst.multi INFO: cancelling 16968765 on provider astro 2024-03-09 20:52:33 proc_lsst.multi:146 [INFO] found job 16968753 in provider astro proc_lsst.multi INFO: found job 16968753 in provider astro 2024-03-09 20:52:33 proc_lsst.multi:201 [INFO] cancelling 16968753 on provider astro proc_lsst.multi INFO: cancelling 16968753 on provider astro 2024-03-09 20:52:33 proc_lsst.multi:146 [INFO] found job 16968757 in provider astro proc_lsst.multi INFO: found job 16968757 in provider astro 2024-03-09 20:52:33 proc_lsst.multi:201 [INFO] cancelling 16968757 on provider astro proc_lsst.multi INFO: cancelling 16968757 on provider astro 2024-03-09 20:52:33 proc_lsst.multi:146 [INFO] found job 16968798 in provider escience_a40 proc_lsst.multi INFO: found job 16968798 in provider escience_a40 2024-03-09 20:52:33 proc_lsst.multi:201 [INFO] cancelling 16968798 on provider escience_a40 proc_lsst.multi INFO: cancelling 16968798 on provider escience_a40 2024-03-09 20:52:33 proc_lsst.multi:146 [INFO] found job 16968813 in provider escience_a40 proc_lsst.multi INFO: found job 16968813 in provider escience_a40 2024-03-09 20:52:33 proc_lsst.multi:201 [INFO] cancelling 16968813 on provider escience_a40 proc_lsst.multi INFO: cancelling 16968813 on provider escience_a40 2024-03-09 20:52:34 proc_lsst.multi:146 [INFO] found job 16968799 in provider escience_a40 proc_lsst.multi INFO: found job 16968799 in provider escience_a40 2024-03-09 20:52:34 proc_lsst.multi:201 [INFO] cancelling 16968799 on provider escience_a40 proc_lsst.multi INFO: cancelling 16968799 on provider escience_a40 2024-03-09 20:52:34 proc_lsst.multi:146 [INFO] found job 16968780 in provider escience_rtx6k proc_lsst.multi INFO: found job 16968780 in provider escience_rtx6k 2024-03-09 20:52:34 proc_lsst.multi:201 [INFO] cancelling 16968780 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16968780 on provider escience_rtx6k 2024-03-09 20:52:34 proc_lsst.multi:146 [INFO] found job 16968784 in provider escience_rtx6k proc_lsst.multi INFO: found job 16968784 in provider escience_rtx6k 2024-03-09 20:52:34 proc_lsst.multi:201 [INFO] cancelling 16968784 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16968784 on provider escience_rtx6k 2024-03-09 20:52:34 proc_lsst.multi:146 [INFO] found job 16968783 in provider escience_rtx6k proc_lsst.multi INFO: found job 16968783 in provider escience_rtx6k 2024-03-09 20:52:34 proc_lsst.multi:201 [INFO] cancelling 16968783 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16968783 on provider escience_rtx6k 2024-03-09 20:52:34 proc_lsst.multi:146 [INFO] found job 16968782 in provider escience_rtx6k proc_lsst.multi INFO: found job 16968782 in provider escience_rtx6k 2024-03-09 20:52:34 proc_lsst.multi:201 [INFO] cancelling 16968782 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16968782 on provider escience_rtx6k 2024-03-09 20:52:35 proc_lsst.multi:146 [INFO] found job 16968751 in provider astro proc_lsst.multi INFO: found job 16968751 in provider astro 2024-03-09 20:52:35 proc_lsst.multi:201 [INFO] cancelling 16968751 on provider astro proc_lsst.multi INFO: cancelling 16968751 on provider astro 2024-03-09 20:52:35 proc_lsst.multi:146 [INFO] found job 16968781 in provider escience_rtx6k proc_lsst.multi INFO: found job 16968781 in provider escience_rtx6k 2024-03-09 20:52:35 proc_lsst.multi:201 [INFO] cancelling 16968781 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16968781 on provider escience_rtx6k 2024-03-09 20:52:35 proc_lsst.multi:146 [INFO] found job 16968752 in provider astro proc_lsst.multi INFO: found job 16968752 in provider astro 2024-03-09 20:52:35 proc_lsst.multi:201 [INFO] cancelling 16968752 on provider astro proc_lsst.multi INFO: cancelling 16968752 on provider astro 2024-03-09 20:52:35 proc_lsst.multi:146 [INFO] found job 16968750 in provider astro proc_lsst.multi INFO: found job 16968750 in provider astro 2024-03-09 20:52:35 proc_lsst.multi:201 [INFO] cancelling 16968750 on provider astro proc_lsst.multi INFO: cancelling 16968750 on provider astro 2024-03-09 20:52:35 proc_lsst.multi:146 [INFO] found job 16968829 in provider checkpoint proc_lsst.multi INFO: found job 16968829 in provider checkpoint 2024-03-09 20:52:35 proc_lsst.multi:201 [INFO] cancelling 16968829 on provider checkpoint proc_lsst.multi INFO: cancelling 16968829 on provider checkpoint 2024-03-09 20:52:35 proc_lsst.multi:146 [INFO] found job 16968826 in provider checkpoint proc_lsst.multi INFO: found job 16968826 in provider checkpoint 2024-03-09 20:52:35 proc_lsst.multi:201 [INFO] cancelling 16968826 on provider checkpoint proc_lsst.multi INFO: cancelling 16968826 on provider checkpoint 2024-03-09 20:52:36 proc_lsst.multi:146 [INFO] found job 16968828 in provider checkpoint proc_lsst.multi INFO: found job 16968828 in provider checkpoint 2024-03-09 20:52:36 proc_lsst.multi:201 [INFO] cancelling 16968828 on provider checkpoint proc_lsst.multi INFO: cancelling 16968828 on provider checkpoint 2024-03-09 20:52:36 proc_lsst.multi:146 [INFO] found job 16968830 in provider checkpoint proc_lsst.multi INFO: found job 16968830 in provider checkpoint 2024-03-09 20:52:36 proc_lsst.multi:201 [INFO] cancelling 16968830 on provider checkpoint proc_lsst.multi INFO: cancelling 16968830 on provider checkpoint 2024-03-09 20:52:36 proc_lsst.multi:146 [INFO] found job 16968833 in provider checkpoint proc_lsst.multi INFO: found job 16968833 in provider checkpoint 2024-03-09 20:52:36 proc_lsst.multi:201 [INFO] cancelling 16968833 on provider checkpoint proc_lsst.multi INFO: cancelling 16968833 on provider checkpoint 2024-03-09 20:52:36 proc_lsst.multi:146 [INFO] found job 16968832 in provider checkpoint proc_lsst.multi INFO: found job 16968832 in provider checkpoint 2024-03-09 20:52:36 proc_lsst.multi:201 [INFO] cancelling 16968832 on provider checkpoint proc_lsst.multi INFO: cancelling 16968832 on provider checkpoint parsl.dataflow.dflow INFO: Shutting down executor multi 2024-03-09 20:52:36 proc_lsst.multi:40 [INFO] Cancelling all provider resources proc_lsst.multi INFO: Cancelling all provider resources 2024-03-09 20:52:36 proc_lsst.multi:47 [INFO] new jobs since last cancel ['16968814', '16968815', '16968816', '16968817', '16968818', '16968819', '16968823', '16968824', '16968825', '16968826', '16968827', '16968828', '16968829', '16968830', '16968831', '16968832', '16968833', '16968834', '16968835', '16968836', '16968837', '16968838', '16968839', '16968840', '16968841', '16968842', '16968843', '16968844', '16968845', '16968846', '16968847', '16968848', '16968849', '16968850', '16968851', '16968852', '16968853', '16968750', '16968751', '16968752', '16968753', '16968754', '16968755', '16968756', '16968757', '16968758', '16968759', '16968760', '16968761', '16968762', '16968763', '16968764', '16968765', '16968766', '16968767', '16968768', '16968769', '16968770', '16968771', '16968772', '16968773', '16968774', '16968775', '16968776', '16968777', '16968778', '16968779', '16968785', '16968786', '16968787', '16968788', '16968789', '16968790', '16968791', '16968792', '16968793', '16968794', '16968795', '16968796', '16968797', '16968798', '16968799', '16968800', '16968805', '16968811', '16968812', '16968813', '16968780', '16968781', '16968782', '16968783', '16968784', '35332'] proc_lsst.multi INFO: new jobs since last cancel ['16968814', '16968815', '16968816', '16968817', '16968818', '16968819', '16968823', '16968824', '16968825', '16968826', '16968827', '16968828', '16968829', '16968830', '16968831', '16968832', '16968833', '16968834', '16968835', '16968836', '16968837', '16968838', '16968839', '16968840', '16968841', '16968842', '16968843', '16968844', '16968845', '16968846', '16968847', '16968848', '16968849', '16968850', '16968851', '16968852', '16968853', '16968750', '16968751', '16968752', '16968753', '16968754', '16968755', '16968756', '16968757', '16968758', '16968759', '16968760', '16968761', '16968762', '16968763', '16968764', '16968765', '16968766', '16968767', '16968768', '16968769', '16968770', '16968771', '16968772', '16968773', '16968774', '16968775', '16968776', '16968777', '16968778', '16968779', '16968785', '16968786', '16968787', '16968788', '16968789', '16968790', '16968791', '16968792', '16968793', '16968794', '16968795', '16968796', '16968797', '16968798', '16968799', '16968800', '16968805', '16968811', '16968812', '16968813', '16968780', '16968781', '16968782', '16968783', '16968784', '35332'] 2024-03-09 20:52:36 proc_lsst.multi:146 [INFO] found job 16968814 in provider checkpoint proc_lsst.multi INFO: found job 16968814 in provider checkpoint 2024-03-09 20:52:36 proc_lsst.multi:201 [INFO] cancelling 16968814 on provider checkpoint proc_lsst.multi INFO: cancelling 16968814 on provider checkpoint 2024-03-09 20:52:36 proc_lsst.multi:146 [INFO] found job 16968815 in provider checkpoint proc_lsst.multi INFO: found job 16968815 in provider checkpoint 2024-03-09 20:52:36 proc_lsst.multi:201 [INFO] cancelling 16968815 on provider checkpoint proc_lsst.multi INFO: cancelling 16968815 on provider checkpoint 2024-03-09 20:52:37 proc_lsst.multi:146 [INFO] found job 16968816 in provider checkpoint proc_lsst.multi INFO: found job 16968816 in provider checkpoint 2024-03-09 20:52:37 proc_lsst.multi:201 [INFO] cancelling 16968816 on provider checkpoint proc_lsst.multi INFO: cancelling 16968816 on provider checkpoint 2024-03-09 20:52:38 proc_lsst.multi:146 [INFO] found job 16968817 in provider checkpoint proc_lsst.multi INFO: found job 16968817 in provider checkpoint 2024-03-09 20:52:38 proc_lsst.multi:201 [INFO] cancelling 16968817 on provider checkpoint proc_lsst.multi INFO: cancelling 16968817 on provider checkpoint 2024-03-09 20:52:38 proc_lsst.multi:146 [INFO] found job 16968818 in provider checkpoint proc_lsst.multi INFO: found job 16968818 in provider checkpoint 2024-03-09 20:52:38 proc_lsst.multi:201 [INFO] cancelling 16968818 on provider checkpoint proc_lsst.multi INFO: cancelling 16968818 on provider checkpoint 2024-03-09 20:52:39 proc_lsst.multi:146 [INFO] found job 16968819 in provider checkpoint proc_lsst.multi INFO: found job 16968819 in provider checkpoint 2024-03-09 20:52:39 proc_lsst.multi:201 [INFO] cancelling 16968819 on provider checkpoint proc_lsst.multi INFO: cancelling 16968819 on provider checkpoint 2024-03-09 20:52:39 proc_lsst.multi:146 [INFO] found job 16968823 in provider checkpoint proc_lsst.multi INFO: found job 16968823 in provider checkpoint 2024-03-09 20:52:39 proc_lsst.multi:201 [INFO] cancelling 16968823 on provider checkpoint proc_lsst.multi INFO: cancelling 16968823 on provider checkpoint 2024-03-09 20:52:39 proc_lsst.multi:146 [INFO] found job 16968824 in provider checkpoint proc_lsst.multi INFO: found job 16968824 in provider checkpoint 2024-03-09 20:52:39 proc_lsst.multi:201 [INFO] cancelling 16968824 on provider checkpoint proc_lsst.multi INFO: cancelling 16968824 on provider checkpoint 2024-03-09 20:52:39 proc_lsst.multi:146 [INFO] found job 16968825 in provider checkpoint proc_lsst.multi INFO: found job 16968825 in provider checkpoint 2024-03-09 20:52:39 proc_lsst.multi:201 [INFO] cancelling 16968825 on provider checkpoint proc_lsst.multi INFO: cancelling 16968825 on provider checkpoint 2024-03-09 20:52:40 proc_lsst.multi:146 [INFO] found job 16968826 in provider checkpoint proc_lsst.multi INFO: found job 16968826 in provider checkpoint 2024-03-09 20:52:40 proc_lsst.multi:201 [INFO] cancelling 16968826 on provider checkpoint proc_lsst.multi INFO: cancelling 16968826 on provider checkpoint 2024-03-09 20:52:40 proc_lsst.multi:146 [INFO] found job 16968827 in provider checkpoint proc_lsst.multi INFO: found job 16968827 in provider checkpoint 2024-03-09 20:52:40 proc_lsst.multi:201 [INFO] cancelling 16968827 on provider checkpoint proc_lsst.multi INFO: cancelling 16968827 on provider checkpoint 2024-03-09 20:52:40 proc_lsst.multi:146 [INFO] found job 16968828 in provider checkpoint proc_lsst.multi INFO: found job 16968828 in provider checkpoint 2024-03-09 20:52:40 proc_lsst.multi:201 [INFO] cancelling 16968828 on provider checkpoint proc_lsst.multi INFO: cancelling 16968828 on provider checkpoint 2024-03-09 20:52:40 proc_lsst.multi:146 [INFO] found job 16968829 in provider checkpoint proc_lsst.multi INFO: found job 16968829 in provider checkpoint 2024-03-09 20:52:40 proc_lsst.multi:201 [INFO] cancelling 16968829 on provider checkpoint proc_lsst.multi INFO: cancelling 16968829 on provider checkpoint 2024-03-09 20:52:40 proc_lsst.multi:146 [INFO] found job 16968830 in provider checkpoint proc_lsst.multi INFO: found job 16968830 in provider checkpoint 2024-03-09 20:52:40 proc_lsst.multi:201 [INFO] cancelling 16968830 on provider checkpoint proc_lsst.multi INFO: cancelling 16968830 on provider checkpoint 2024-03-09 20:52:41 proc_lsst.multi:146 [INFO] found job 16968831 in provider checkpoint proc_lsst.multi INFO: found job 16968831 in provider checkpoint 2024-03-09 20:52:41 proc_lsst.multi:201 [INFO] cancelling 16968831 on provider checkpoint proc_lsst.multi INFO: cancelling 16968831 on provider checkpoint 2024-03-09 20:52:41 proc_lsst.multi:146 [INFO] found job 16968832 in provider checkpoint proc_lsst.multi INFO: found job 16968832 in provider checkpoint 2024-03-09 20:52:41 proc_lsst.multi:201 [INFO] cancelling 16968832 on provider checkpoint proc_lsst.multi INFO: cancelling 16968832 on provider checkpoint 2024-03-09 20:52:41 proc_lsst.multi:146 [INFO] found job 16968833 in provider checkpoint proc_lsst.multi INFO: found job 16968833 in provider checkpoint 2024-03-09 20:52:41 proc_lsst.multi:201 [INFO] cancelling 16968833 on provider checkpoint proc_lsst.multi INFO: cancelling 16968833 on provider checkpoint 2024-03-09 20:52:41 proc_lsst.multi:146 [INFO] found job 16968834 in provider checkpoint proc_lsst.multi INFO: found job 16968834 in provider checkpoint 2024-03-09 20:52:41 proc_lsst.multi:201 [INFO] cancelling 16968834 on provider checkpoint proc_lsst.multi INFO: cancelling 16968834 on provider checkpoint 2024-03-09 20:52:41 proc_lsst.multi:146 [INFO] found job 16968835 in provider checkpoint proc_lsst.multi INFO: found job 16968835 in provider checkpoint 2024-03-09 20:52:41 proc_lsst.multi:201 [INFO] cancelling 16968835 on provider checkpoint proc_lsst.multi INFO: cancelling 16968835 on provider checkpoint 2024-03-09 20:52:41 proc_lsst.multi:146 [INFO] found job 16968836 in provider checkpoint proc_lsst.multi INFO: found job 16968836 in provider checkpoint 2024-03-09 20:52:41 proc_lsst.multi:201 [INFO] cancelling 16968836 on provider checkpoint proc_lsst.multi INFO: cancelling 16968836 on provider checkpoint 2024-03-09 20:52:42 proc_lsst.multi:146 [INFO] found job 16968837 in provider checkpoint proc_lsst.multi INFO: found job 16968837 in provider checkpoint 2024-03-09 20:52:42 proc_lsst.multi:201 [INFO] cancelling 16968837 on provider checkpoint proc_lsst.multi INFO: cancelling 16968837 on provider checkpoint 2024-03-09 20:52:42 proc_lsst.multi:146 [INFO] found job 16968838 in provider checkpoint proc_lsst.multi INFO: found job 16968838 in provider checkpoint 2024-03-09 20:52:42 proc_lsst.multi:201 [INFO] cancelling 16968838 on provider checkpoint proc_lsst.multi INFO: cancelling 16968838 on provider checkpoint 2024-03-09 20:52:42 proc_lsst.multi:146 [INFO] found job 16968839 in provider checkpoint proc_lsst.multi INFO: found job 16968839 in provider checkpoint 2024-03-09 20:52:42 proc_lsst.multi:201 [INFO] cancelling 16968839 on provider checkpoint proc_lsst.multi INFO: cancelling 16968839 on provider checkpoint 2024-03-09 20:52:42 proc_lsst.multi:146 [INFO] found job 16968840 in provider checkpoint proc_lsst.multi INFO: found job 16968840 in provider checkpoint 2024-03-09 20:52:42 proc_lsst.multi:201 [INFO] cancelling 16968840 on provider checkpoint proc_lsst.multi INFO: cancelling 16968840 on provider checkpoint 2024-03-09 20:52:42 proc_lsst.multi:146 [INFO] found job 16968841 in provider checkpoint proc_lsst.multi INFO: found job 16968841 in provider checkpoint 2024-03-09 20:52:42 proc_lsst.multi:201 [INFO] cancelling 16968841 on provider checkpoint proc_lsst.multi INFO: cancelling 16968841 on provider checkpoint 2024-03-09 20:52:42 proc_lsst.multi:146 [INFO] found job 16968842 in provider checkpoint proc_lsst.multi INFO: found job 16968842 in provider checkpoint 2024-03-09 20:52:42 proc_lsst.multi:201 [INFO] cancelling 16968842 on provider checkpoint proc_lsst.multi INFO: cancelling 16968842 on provider checkpoint 2024-03-09 20:52:43 proc_lsst.multi:146 [INFO] found job 16968843 in provider checkpoint proc_lsst.multi INFO: found job 16968843 in provider checkpoint 2024-03-09 20:52:43 proc_lsst.multi:201 [INFO] cancelling 16968843 on provider checkpoint proc_lsst.multi INFO: cancelling 16968843 on provider checkpoint 2024-03-09 20:52:43 proc_lsst.multi:146 [INFO] found job 16968844 in provider checkpoint proc_lsst.multi INFO: found job 16968844 in provider checkpoint 2024-03-09 20:52:43 proc_lsst.multi:201 [INFO] cancelling 16968844 on provider checkpoint proc_lsst.multi INFO: cancelling 16968844 on provider checkpoint 2024-03-09 20:52:43 proc_lsst.multi:146 [INFO] found job 16968845 in provider checkpoint proc_lsst.multi INFO: found job 16968845 in provider checkpoint 2024-03-09 20:52:43 proc_lsst.multi:201 [INFO] cancelling 16968845 on provider checkpoint proc_lsst.multi INFO: cancelling 16968845 on provider checkpoint 2024-03-09 20:52:43 proc_lsst.multi:146 [INFO] found job 16968846 in provider checkpoint proc_lsst.multi INFO: found job 16968846 in provider checkpoint 2024-03-09 20:52:43 proc_lsst.multi:201 [INFO] cancelling 16968846 on provider checkpoint proc_lsst.multi INFO: cancelling 16968846 on provider checkpoint 2024-03-09 20:52:43 proc_lsst.multi:146 [INFO] found job 16968847 in provider checkpoint proc_lsst.multi INFO: found job 16968847 in provider checkpoint 2024-03-09 20:52:43 proc_lsst.multi:201 [INFO] cancelling 16968847 on provider checkpoint proc_lsst.multi INFO: cancelling 16968847 on provider checkpoint 2024-03-09 20:52:43 proc_lsst.multi:146 [INFO] found job 16968848 in provider checkpoint proc_lsst.multi INFO: found job 16968848 in provider checkpoint 2024-03-09 20:52:43 proc_lsst.multi:201 [INFO] cancelling 16968848 on provider checkpoint proc_lsst.multi INFO: cancelling 16968848 on provider checkpoint 2024-03-09 20:52:43 proc_lsst.multi:146 [INFO] found job 16968849 in provider checkpoint proc_lsst.multi INFO: found job 16968849 in provider checkpoint 2024-03-09 20:52:43 proc_lsst.multi:201 [INFO] cancelling 16968849 on provider checkpoint proc_lsst.multi INFO: cancelling 16968849 on provider checkpoint 2024-03-09 20:52:44 proc_lsst.multi:146 [INFO] found job 16968850 in provider checkpoint proc_lsst.multi INFO: found job 16968850 in provider checkpoint 2024-03-09 20:52:44 proc_lsst.multi:201 [INFO] cancelling 16968850 on provider checkpoint proc_lsst.multi INFO: cancelling 16968850 on provider checkpoint 2024-03-09 20:52:44 proc_lsst.multi:146 [INFO] found job 16968851 in provider checkpoint proc_lsst.multi INFO: found job 16968851 in provider checkpoint 2024-03-09 20:52:44 proc_lsst.multi:201 [INFO] cancelling 16968851 on provider checkpoint proc_lsst.multi INFO: cancelling 16968851 on provider checkpoint 2024-03-09 20:52:44 proc_lsst.multi:146 [INFO] found job 16968852 in provider checkpoint proc_lsst.multi INFO: found job 16968852 in provider checkpoint 2024-03-09 20:52:44 proc_lsst.multi:201 [INFO] cancelling 16968852 on provider checkpoint proc_lsst.multi INFO: cancelling 16968852 on provider checkpoint 2024-03-09 20:52:44 proc_lsst.multi:146 [INFO] found job 16968853 in provider checkpoint proc_lsst.multi INFO: found job 16968853 in provider checkpoint 2024-03-09 20:52:44 proc_lsst.multi:201 [INFO] cancelling 16968853 on provider checkpoint proc_lsst.multi INFO: cancelling 16968853 on provider checkpoint 2024-03-09 20:52:44 proc_lsst.multi:146 [INFO] found job 16968750 in provider astro proc_lsst.multi INFO: found job 16968750 in provider astro 2024-03-09 20:52:44 proc_lsst.multi:201 [INFO] cancelling 16968750 on provider astro proc_lsst.multi INFO: cancelling 16968750 on provider astro 2024-03-09 20:52:44 proc_lsst.multi:146 [INFO] found job 16968751 in provider astro proc_lsst.multi INFO: found job 16968751 in provider astro 2024-03-09 20:52:44 proc_lsst.multi:201 [INFO] cancelling 16968751 on provider astro proc_lsst.multi INFO: cancelling 16968751 on provider astro 2024-03-09 20:52:45 proc_lsst.multi:146 [INFO] found job 16968752 in provider astro proc_lsst.multi INFO: found job 16968752 in provider astro 2024-03-09 20:52:45 proc_lsst.multi:201 [INFO] cancelling 16968752 on provider astro proc_lsst.multi INFO: cancelling 16968752 on provider astro 2024-03-09 20:52:45 proc_lsst.multi:146 [INFO] found job 16968753 in provider astro proc_lsst.multi INFO: found job 16968753 in provider astro 2024-03-09 20:52:45 proc_lsst.multi:201 [INFO] cancelling 16968753 on provider astro proc_lsst.multi INFO: cancelling 16968753 on provider astro 2024-03-09 20:52:45 proc_lsst.multi:146 [INFO] found job 16968754 in provider astro proc_lsst.multi INFO: found job 16968754 in provider astro 2024-03-09 20:52:45 proc_lsst.multi:201 [INFO] cancelling 16968754 on provider astro proc_lsst.multi INFO: cancelling 16968754 on provider astro 2024-03-09 20:52:45 proc_lsst.multi:146 [INFO] found job 16968755 in provider astro proc_lsst.multi INFO: found job 16968755 in provider astro 2024-03-09 20:52:45 proc_lsst.multi:201 [INFO] cancelling 16968755 on provider astro proc_lsst.multi INFO: cancelling 16968755 on provider astro 2024-03-09 20:52:45 proc_lsst.multi:146 [INFO] found job 16968756 in provider astro proc_lsst.multi INFO: found job 16968756 in provider astro 2024-03-09 20:52:45 proc_lsst.multi:201 [INFO] cancelling 16968756 on provider astro proc_lsst.multi INFO: cancelling 16968756 on provider astro 2024-03-09 20:52:45 proc_lsst.multi:146 [INFO] found job 16968757 in provider astro proc_lsst.multi INFO: found job 16968757 in provider astro 2024-03-09 20:52:45 proc_lsst.multi:201 [INFO] cancelling 16968757 on provider astro proc_lsst.multi INFO: cancelling 16968757 on provider astro 2024-03-09 20:52:46 proc_lsst.multi:146 [INFO] found job 16968758 in provider astro proc_lsst.multi INFO: found job 16968758 in provider astro 2024-03-09 20:52:46 proc_lsst.multi:201 [INFO] cancelling 16968758 on provider astro proc_lsst.multi INFO: cancelling 16968758 on provider astro 2024-03-09 20:52:46 proc_lsst.multi:146 [INFO] found job 16968759 in provider astro proc_lsst.multi INFO: found job 16968759 in provider astro 2024-03-09 20:52:46 proc_lsst.multi:201 [INFO] cancelling 16968759 on provider astro proc_lsst.multi INFO: cancelling 16968759 on provider astro 2024-03-09 20:52:46 proc_lsst.multi:146 [INFO] found job 16968760 in provider astro proc_lsst.multi INFO: found job 16968760 in provider astro 2024-03-09 20:52:46 proc_lsst.multi:201 [INFO] cancelling 16968760 on provider astro proc_lsst.multi INFO: cancelling 16968760 on provider astro 2024-03-09 20:52:46 proc_lsst.multi:146 [INFO] found job 16968761 in provider astro proc_lsst.multi INFO: found job 16968761 in provider astro 2024-03-09 20:52:46 proc_lsst.multi:201 [INFO] cancelling 16968761 on provider astro proc_lsst.multi INFO: cancelling 16968761 on provider astro 2024-03-09 20:52:46 proc_lsst.multi:146 [INFO] found job 16968762 in provider astro proc_lsst.multi INFO: found job 16968762 in provider astro 2024-03-09 20:52:46 proc_lsst.multi:201 [INFO] cancelling 16968762 on provider astro proc_lsst.multi INFO: cancelling 16968762 on provider astro 2024-03-09 20:52:46 proc_lsst.multi:146 [INFO] found job 16968763 in provider astro proc_lsst.multi INFO: found job 16968763 in provider astro 2024-03-09 20:52:46 proc_lsst.multi:201 [INFO] cancelling 16968763 on provider astro proc_lsst.multi INFO: cancelling 16968763 on provider astro 2024-03-09 20:52:47 proc_lsst.multi:146 [INFO] found job 16968764 in provider astro proc_lsst.multi INFO: found job 16968764 in provider astro 2024-03-09 20:52:47 proc_lsst.multi:201 [INFO] cancelling 16968764 on provider astro proc_lsst.multi INFO: cancelling 16968764 on provider astro 2024-03-09 20:52:47 proc_lsst.multi:146 [INFO] found job 16968765 in provider astro proc_lsst.multi INFO: found job 16968765 in provider astro 2024-03-09 20:52:47 proc_lsst.multi:201 [INFO] cancelling 16968765 on provider astro proc_lsst.multi INFO: cancelling 16968765 on provider astro 2024-03-09 20:52:47 proc_lsst.multi:146 [INFO] found job 16968766 in provider astro proc_lsst.multi INFO: found job 16968766 in provider astro 2024-03-09 20:52:47 proc_lsst.multi:201 [INFO] cancelling 16968766 on provider astro proc_lsst.multi INFO: cancelling 16968766 on provider astro 2024-03-09 20:52:47 proc_lsst.multi:146 [INFO] found job 16968767 in provider astro proc_lsst.multi INFO: found job 16968767 in provider astro 2024-03-09 20:52:47 proc_lsst.multi:201 [INFO] cancelling 16968767 on provider astro proc_lsst.multi INFO: cancelling 16968767 on provider astro 2024-03-09 20:52:47 proc_lsst.multi:146 [INFO] found job 16968768 in provider astro proc_lsst.multi INFO: found job 16968768 in provider astro 2024-03-09 20:52:47 proc_lsst.multi:201 [INFO] cancelling 16968768 on provider astro proc_lsst.multi INFO: cancelling 16968768 on provider astro 2024-03-09 20:52:47 proc_lsst.multi:146 [INFO] found job 16968769 in provider astro proc_lsst.multi INFO: found job 16968769 in provider astro 2024-03-09 20:52:47 proc_lsst.multi:201 [INFO] cancelling 16968769 on provider astro proc_lsst.multi INFO: cancelling 16968769 on provider astro 2024-03-09 20:52:48 proc_lsst.multi:146 [INFO] found job 16968770 in provider astro proc_lsst.multi INFO: found job 16968770 in provider astro 2024-03-09 20:52:48 proc_lsst.multi:201 [INFO] cancelling 16968770 on provider astro proc_lsst.multi INFO: cancelling 16968770 on provider astro 2024-03-09 20:52:48 proc_lsst.multi:146 [INFO] found job 16968771 in provider astro proc_lsst.multi INFO: found job 16968771 in provider astro 2024-03-09 20:52:48 proc_lsst.multi:201 [INFO] cancelling 16968771 on provider astro proc_lsst.multi INFO: cancelling 16968771 on provider astro 2024-03-09 20:52:48 proc_lsst.multi:146 [INFO] found job 16968772 in provider astro proc_lsst.multi INFO: found job 16968772 in provider astro 2024-03-09 20:52:48 proc_lsst.multi:201 [INFO] cancelling 16968772 on provider astro proc_lsst.multi INFO: cancelling 16968772 on provider astro 2024-03-09 20:52:48 proc_lsst.multi:146 [INFO] found job 16968773 in provider astro proc_lsst.multi INFO: found job 16968773 in provider astro 2024-03-09 20:52:48 proc_lsst.multi:201 [INFO] cancelling 16968773 on provider astro proc_lsst.multi INFO: cancelling 16968773 on provider astro 2024-03-09 20:52:48 proc_lsst.multi:146 [INFO] found job 16968774 in provider astro proc_lsst.multi INFO: found job 16968774 in provider astro 2024-03-09 20:52:48 proc_lsst.multi:201 [INFO] cancelling 16968774 on provider astro proc_lsst.multi INFO: cancelling 16968774 on provider astro 2024-03-09 20:52:48 proc_lsst.multi:146 [INFO] found job 16968775 in provider astro proc_lsst.multi INFO: found job 16968775 in provider astro 2024-03-09 20:52:48 proc_lsst.multi:201 [INFO] cancelling 16968775 on provider astro proc_lsst.multi INFO: cancelling 16968775 on provider astro 2024-03-09 20:52:48 proc_lsst.multi:146 [INFO] found job 16968776 in provider astro proc_lsst.multi INFO: found job 16968776 in provider astro 2024-03-09 20:52:48 proc_lsst.multi:201 [INFO] cancelling 16968776 on provider astro proc_lsst.multi INFO: cancelling 16968776 on provider astro 2024-03-09 20:52:49 proc_lsst.multi:146 [INFO] found job 16968777 in provider astro proc_lsst.multi INFO: found job 16968777 in provider astro 2024-03-09 20:52:49 proc_lsst.multi:201 [INFO] cancelling 16968777 on provider astro proc_lsst.multi INFO: cancelling 16968777 on provider astro 2024-03-09 20:52:49 proc_lsst.multi:146 [INFO] found job 16968778 in provider astro proc_lsst.multi INFO: found job 16968778 in provider astro 2024-03-09 20:52:49 proc_lsst.multi:201 [INFO] cancelling 16968778 on provider astro proc_lsst.multi INFO: cancelling 16968778 on provider astro 2024-03-09 20:52:49 proc_lsst.multi:146 [INFO] found job 16968779 in provider astro proc_lsst.multi INFO: found job 16968779 in provider astro 2024-03-09 20:52:49 proc_lsst.multi:201 [INFO] cancelling 16968779 on provider astro proc_lsst.multi INFO: cancelling 16968779 on provider astro 2024-03-09 20:52:49 proc_lsst.multi:146 [INFO] found job 16968785 in provider escience_a40 proc_lsst.multi INFO: found job 16968785 in provider escience_a40 2024-03-09 20:52:49 proc_lsst.multi:201 [INFO] cancelling 16968785 on provider escience_a40 proc_lsst.multi INFO: cancelling 16968785 on provider escience_a40 2024-03-09 20:52:49 proc_lsst.multi:146 [INFO] found job 16968786 in provider escience_a40 proc_lsst.multi INFO: found job 16968786 in provider escience_a40 2024-03-09 20:52:49 proc_lsst.multi:201 [INFO] cancelling 16968786 on provider escience_a40 proc_lsst.multi INFO: cancelling 16968786 on provider escience_a40 2024-03-09 20:52:49 proc_lsst.multi:146 [INFO] found job 16968787 in provider escience_a40 proc_lsst.multi INFO: found job 16968787 in provider escience_a40 2024-03-09 20:52:49 proc_lsst.multi:201 [INFO] cancelling 16968787 on provider escience_a40 proc_lsst.multi INFO: cancelling 16968787 on provider escience_a40 2024-03-09 20:52:50 proc_lsst.multi:146 [INFO] found job 16968788 in provider escience_a40 proc_lsst.multi INFO: found job 16968788 in provider escience_a40 2024-03-09 20:52:50 proc_lsst.multi:201 [INFO] cancelling 16968788 on provider escience_a40 proc_lsst.multi INFO: cancelling 16968788 on provider escience_a40 2024-03-09 20:52:50 proc_lsst.multi:146 [INFO] found job 16968789 in provider escience_a40 proc_lsst.multi INFO: found job 16968789 in provider escience_a40 2024-03-09 20:52:50 proc_lsst.multi:201 [INFO] cancelling 16968789 on provider escience_a40 proc_lsst.multi INFO: cancelling 16968789 on provider escience_a40 2024-03-09 20:52:50 proc_lsst.multi:146 [INFO] found job 16968790 in provider escience_a40 proc_lsst.multi INFO: found job 16968790 in provider escience_a40 2024-03-09 20:52:50 proc_lsst.multi:201 [INFO] cancelling 16968790 on provider escience_a40 proc_lsst.multi INFO: cancelling 16968790 on provider escience_a40 2024-03-09 20:52:50 proc_lsst.multi:146 [INFO] found job 16968791 in provider escience_a40 proc_lsst.multi INFO: found job 16968791 in provider escience_a40 2024-03-09 20:52:50 proc_lsst.multi:201 [INFO] cancelling 16968791 on provider escience_a40 proc_lsst.multi INFO: cancelling 16968791 on provider escience_a40 2024-03-09 20:52:50 proc_lsst.multi:146 [INFO] found job 16968792 in provider escience_a40 proc_lsst.multi INFO: found job 16968792 in provider escience_a40 2024-03-09 20:52:50 proc_lsst.multi:201 [INFO] cancelling 16968792 on provider escience_a40 proc_lsst.multi INFO: cancelling 16968792 on provider escience_a40 2024-03-09 20:52:50 proc_lsst.multi:146 [INFO] found job 16968793 in provider escience_a40 proc_lsst.multi INFO: found job 16968793 in provider escience_a40 2024-03-09 20:52:50 proc_lsst.multi:201 [INFO] cancelling 16968793 on provider escience_a40 proc_lsst.multi INFO: cancelling 16968793 on provider escience_a40 2024-03-09 20:52:50 proc_lsst.multi:146 [INFO] found job 16968794 in provider escience_a40 proc_lsst.multi INFO: found job 16968794 in provider escience_a40 2024-03-09 20:52:50 proc_lsst.multi:201 [INFO] cancelling 16968794 on provider escience_a40 proc_lsst.multi INFO: cancelling 16968794 on provider escience_a40 2024-03-09 20:52:51 proc_lsst.multi:146 [INFO] found job 16968795 in provider escience_a40 proc_lsst.multi INFO: found job 16968795 in provider escience_a40 2024-03-09 20:52:51 proc_lsst.multi:201 [INFO] cancelling 16968795 on provider escience_a40 proc_lsst.multi INFO: cancelling 16968795 on provider escience_a40 2024-03-09 20:52:51 proc_lsst.multi:146 [INFO] found job 16968796 in provider escience_a40 proc_lsst.multi INFO: found job 16968796 in provider escience_a40 2024-03-09 20:52:51 proc_lsst.multi:201 [INFO] cancelling 16968796 on provider escience_a40 proc_lsst.multi INFO: cancelling 16968796 on provider escience_a40 2024-03-09 20:52:51 proc_lsst.multi:146 [INFO] found job 16968797 in provider escience_a40 proc_lsst.multi INFO: found job 16968797 in provider escience_a40 2024-03-09 20:52:51 proc_lsst.multi:201 [INFO] cancelling 16968797 on provider escience_a40 proc_lsst.multi INFO: cancelling 16968797 on provider escience_a40 2024-03-09 20:52:51 proc_lsst.multi:146 [INFO] found job 16968798 in provider escience_a40 proc_lsst.multi INFO: found job 16968798 in provider escience_a40 2024-03-09 20:52:51 proc_lsst.multi:201 [INFO] cancelling 16968798 on provider escience_a40 proc_lsst.multi INFO: cancelling 16968798 on provider escience_a40 2024-03-09 20:52:51 proc_lsst.multi:146 [INFO] found job 16968799 in provider escience_a40 proc_lsst.multi INFO: found job 16968799 in provider escience_a40 2024-03-09 20:52:51 proc_lsst.multi:201 [INFO] cancelling 16968799 on provider escience_a40 proc_lsst.multi INFO: cancelling 16968799 on provider escience_a40 2024-03-09 20:52:51 proc_lsst.multi:146 [INFO] found job 16968800 in provider escience_a40 proc_lsst.multi INFO: found job 16968800 in provider escience_a40 2024-03-09 20:52:51 proc_lsst.multi:201 [INFO] cancelling 16968800 on provider escience_a40 proc_lsst.multi INFO: cancelling 16968800 on provider escience_a40 2024-03-09 20:52:52 proc_lsst.multi:146 [INFO] found job 16968805 in provider escience_a40 proc_lsst.multi INFO: found job 16968805 in provider escience_a40 2024-03-09 20:52:52 proc_lsst.multi:201 [INFO] cancelling 16968805 on provider escience_a40 proc_lsst.multi INFO: cancelling 16968805 on provider escience_a40 2024-03-09 20:52:52 proc_lsst.multi:146 [INFO] found job 16968811 in provider escience_a40 proc_lsst.multi INFO: found job 16968811 in provider escience_a40 2024-03-09 20:52:52 proc_lsst.multi:201 [INFO] cancelling 16968811 on provider escience_a40 proc_lsst.multi INFO: cancelling 16968811 on provider escience_a40 2024-03-09 20:52:52 proc_lsst.multi:146 [INFO] found job 16968812 in provider escience_a40 proc_lsst.multi INFO: found job 16968812 in provider escience_a40 2024-03-09 20:52:52 proc_lsst.multi:201 [INFO] cancelling 16968812 on provider escience_a40 proc_lsst.multi INFO: cancelling 16968812 on provider escience_a40 2024-03-09 20:52:52 proc_lsst.multi:146 [INFO] found job 16968813 in provider escience_a40 proc_lsst.multi INFO: found job 16968813 in provider escience_a40 2024-03-09 20:52:52 proc_lsst.multi:201 [INFO] cancelling 16968813 on provider escience_a40 proc_lsst.multi INFO: cancelling 16968813 on provider escience_a40 2024-03-09 20:52:52 proc_lsst.multi:146 [INFO] found job 16968780 in provider escience_rtx6k proc_lsst.multi INFO: found job 16968780 in provider escience_rtx6k 2024-03-09 20:52:52 proc_lsst.multi:201 [INFO] cancelling 16968780 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16968780 on provider escience_rtx6k 2024-03-09 20:52:52 proc_lsst.multi:146 [INFO] found job 16968781 in provider escience_rtx6k proc_lsst.multi INFO: found job 16968781 in provider escience_rtx6k 2024-03-09 20:52:52 proc_lsst.multi:201 [INFO] cancelling 16968781 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16968781 on provider escience_rtx6k 2024-03-09 20:52:53 proc_lsst.multi:146 [INFO] found job 16968782 in provider escience_rtx6k proc_lsst.multi INFO: found job 16968782 in provider escience_rtx6k 2024-03-09 20:52:53 proc_lsst.multi:201 [INFO] cancelling 16968782 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16968782 on provider escience_rtx6k 2024-03-09 20:52:53 proc_lsst.multi:146 [INFO] found job 16968783 in provider escience_rtx6k proc_lsst.multi INFO: found job 16968783 in provider escience_rtx6k 2024-03-09 20:52:53 proc_lsst.multi:201 [INFO] cancelling 16968783 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16968783 on provider escience_rtx6k 2024-03-09 20:52:53 proc_lsst.multi:146 [INFO] found job 16968784 in provider escience_rtx6k proc_lsst.multi INFO: found job 16968784 in provider escience_rtx6k 2024-03-09 20:52:53 proc_lsst.multi:201 [INFO] cancelling 16968784 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16968784 on provider escience_rtx6k 2024-03-09 20:52:53 proc_lsst.multi:146 [INFO] found job 35332 in provider local proc_lsst.multi INFO: found job 35332 in provider local 2024-03-09 20:52:53 proc_lsst.multi:201 [INFO] cancelling 35332 on provider local proc_lsst.multi INFO: cancelling 35332 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 35332 parsl.providers.local.local WARNING: Failed to kill PID: 35332 and child processes on local 2024-03-09 20:52:54 proc_lsst.multi:50 [INFO] no new jobs since last cancel, resuming executor shutdown proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt parsl.dataflow.dflow INFO: Shut down executor multi parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete parsl.executors.threads DEBUG: Done with executor shutdown parsl.dataflow.dflow INFO: Shut down executor _parsl_internal parsl.dataflow.dflow INFO: Terminated executors parsl.dataflow.dflow INFO: DFK cleanup complete parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 234.6240 seconds lsst.ctrl.bps.drivers INFO: Run 'DEEP_20190507_A1c_science#step0_20240309T204810Z' submitted for execution with id 'None' lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 234.6359 seconds; current memory usage: 0.277 Gibyte, delta: 0.001 Gibyte, peak delta: 0.003 Gibyte lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 279.4443 seconds; current memory usage: 0.277 Gibyte, delta: 0.138 Gibyte, peak delta: 0.140 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.279 Gibyte (main), 0.279 Gibyte (largest child process) Run Id: None Run Name: DEEP_20190507_A1c_science#step0_20240309T204810Z parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up real 4m42.371s user 1m21.421s sys 0m42.682s