lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process. lsst.ctrl.bps.drivers INFO: Starting submission process lsst.ctrl.bps.drivers INFO: Initializing execution environment lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 19.3911 seconds; current memory usage: 0.178 Gibyte, delta: 0.042 Gibyte, peak delta: 0.054 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.190 Gibyte (main), 0.000 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph) lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.5531 seconds lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.5350 seconds lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/pipeline_orig.qgraph DEEP/20220821/B1m/science#step0/20240307T033310Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/pipeline.qgraph lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 2.5498 seconds lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 8.3937 seconds; current memory usage: 0.270 Gibyte, delta: 0.092 Gibyte, peak delta: 0.080 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.270 Gibyte (main), 0.264 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs) lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0118 seconds; current memory usage: 0.270 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.270 Gibyte (main), 0.264 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 248 cluster(s) lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow) lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20220821_B1m_science#step0_20240307T033310Z' lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 0.6277 seconds; current memory usage: 0.270 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.270 Gibyte (main), 0.264 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 250 job(s) (including final) lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow) parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2 Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses s_addresses.add(address_by_interface(interface)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper retval = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface return socket.inet_ntoa(fcntl.ioctl( ^^^^^^^^^^^^ OSError: [Errno 99] Cannot assign requested address lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 1.8958 seconds; current memory usage: 0.272 Gibyte, delta: 0.002 Gibyte, peak delta: 0.002 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.272 Gibyte (main), 0.264 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting submit stage lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000 parsl.dataflow.dflow INFO: Starting DataFlowKernel with config Config( app_cache=True, checkpoint_files=None, checkpoint_mode='task_exit', checkpoint_period=None, executors=[MultiHighThroughputExecutor()], garbage_collect=True, initialize_logging=True, internal_tasks_max_threads=10, max_idletime=120.0, monitoring=None, retries=1, retry_handler=None, run_dir='runinfo', strategy='simple', usage_tracking=False ) parsl.dataflow.dflow INFO: Parsl version: 2023.06.12 parsl.usage_tracking.usage DEBUG: Tracking status: False parsl.dataflow.dflow INFO: Run id is: 466096f3-b3a0-4f8f-b574-6f7068480ab4 parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name parsl.dataflow.memoization INFO: App caching initialized parsl.dataflow.strategy DEBUG: Scaling strategy: simple parsl.executors.high_throughput.executor DEBUG: Starting queue management thread parsl.executors.high_throughput.executor DEBUG: queue management worker starting parsl.executors.high_throughput.executor DEBUG: Started queue management thread Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z parsl.executors.high_throughput.executor DEBUG: Created management thread: parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id={block_id} --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider: parsl.executors.status_handling INFO: Scaling out by 1 blocks parsl.executors.status_handling INFO: Allocated block ID 0 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0 2024-03-07 03:33:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 2024-03-07 03:33:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:33:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-07 03:33:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:33:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local parsl.providers.local.local DEBUG: Launching in remote mode 2024-03-07 03:33:58 proc_lsst.multi:170 [INFO] [multi] job_id 766 proc_lsst.multi INFO: [multi] job_id 766 2024-03-07 03:33:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:33:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:33:58 proc_lsst.multi:178 [INFO] [multi] provider local accepted submit and returned 766 proc_lsst.multi INFO: [multi] provider local accepted submit and returned 766 parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 766 parsl.dataflow.job_status_poller DEBUG: Adding executor multi parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 0 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 0 has memoization hash 5162d17d39101d83a47b696f7604c1a5 parsl.dataflow.memoization INFO: Task 0 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87aa06980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1 parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/b0e6bdf0-d9d7-4b0d-b39a-56f000229767_isrForCrosstalkSources_1123954_15.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/b0e6bdf0-d9d7-4b0d-b39a-56f000229767_isrForCrosstalkSources_1123954_15.stderr parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1 has memoization hash 660bde7bf87994c1a7b5454bd3fd08c1 parsl.dataflow.memoization INFO: Task 1 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87aa06ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 2 parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/5a11f9c0-d6e9-47ff-a71f-8daa9cefdbed_isrForCrosstalkSources_1123956_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/5a11f9c0-d6e9-47ff-a71f-8daa9cefdbed_isrForCrosstalkSources_1123956_47.stderr parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2 has memoization hash 1dd3f8228bb419af7907dcfb9ab8c113 parsl.dataflow.memoization INFO: Task 2 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87aa06fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 3 parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/ad5ad05d-a107-41d4-bc57-cb9ae32c4931_isrForCrosstalkSources_1123956_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/ad5ad05d-a107-41d4-bc57-cb9ae32c4931_isrForCrosstalkSources_1123956_50.stderr parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3 has memoization hash b7d023f7829da0c7125aa11d06debaf3 parsl.dataflow.memoization INFO: Task 3 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87aa06e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 4 parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/927f30b4-db8e-475f-aeb5-81b93f9594de_isrForCrosstalkSources_1124087_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/927f30b4-db8e-475f-aeb5-81b93f9594de_isrForCrosstalkSources_1124087_42.stderr parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4 has memoization hash 133334d3a88bbcfcfd73d3d3cd288750 parsl.dataflow.memoization INFO: Task 4 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87aa06f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 5 parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/b4252b57-8f2a-4844-94c9-4cb2add53bb7_isrForCrosstalkSources_1124087_8.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/b4252b57-8f2a-4844-94c9-4cb2add53bb7_isrForCrosstalkSources_1124087_8.stderr parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5 has memoization hash c0002a9b7f4fa158ecfd8f315ca96cac parsl.dataflow.memoization INFO: Task 5 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87aa07100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 6 parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/9c743906-f335-4e5e-b6bf-a72c8ab6e2f0_isrForCrosstalkSources_1123958_57.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/9c743906-f335-4e5e-b6bf-a72c8ab6e2f0_isrForCrosstalkSources_1123958_57.stderr parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 6 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 6 has memoization hash 72c9591e84f0e6614fb1b29f0ac4d92b parsl.dataflow.memoization INFO: Task 6 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87aa071a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 7 parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/bcc36014-3fd4-4300-b8f2-6a33823c01d9_isrForCrosstalkSources_1124087_25.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/bcc36014-3fd4-4300-b8f2-6a33823c01d9_isrForCrosstalkSources_1124087_25.stderr parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 7 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 7 has memoization hash 61cff9aa79cc7e795d89205eccb57de9 parsl.dataflow.memoization INFO: Task 7 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87aa062a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 8 parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/436af49a-9ebb-4618-ab8b-0d9f24a1df38_isrForCrosstalkSources_1123956_26.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/436af49a-9ebb-4618-ab8b-0d9f24a1df38_isrForCrosstalkSources_1123956_26.stderr parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 8 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 8 has memoization hash 48ac6f7050a4f9858ac25ca258cca7c7 parsl.dataflow.memoization INFO: Task 8 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87aa06b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 9 parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/891a6879-75a1-4415-946b-a3b67ae88d3f_isrForCrosstalkSources_1123954_36.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/891a6879-75a1-4415-946b-a3b67ae88d3f_isrForCrosstalkSources_1123954_36.stderr parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 9 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 9 has memoization hash 921fc569894e5d47bb18eaa41b1900a7 parsl.dataflow.memoization INFO: Task 9 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87aa072e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 10 parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/eb8b6fa9-29f2-421c-be14-7b07844bd0ee_isrForCrosstalkSources_1123958_4.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/eb8b6fa9-29f2-421c-be14-7b07844bd0ee_isrForCrosstalkSources_1123958_4.stderr parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 10 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 10 has memoization hash 831abaf6e2367b82e0d006fb984c10b3 parsl.dataflow.memoization INFO: Task 10 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87aa07060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 11 parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/a2e2db14-4b39-4851-80b2-119fd4283b5d_isrForCrosstalkSources_1123956_42.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/a2e2db14-4b39-4851-80b2-119fd4283b5d_isrForCrosstalkSources_1123956_42.stderr parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 11 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 11 has memoization hash 21138e331816e7f585ca978d73fd4a53 parsl.dataflow.memoization INFO: Task 11 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87aa07420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 12 parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/6f46e39e-685a-4b1a-a5ef-7a10394dd121_isrForCrosstalkSources_1124087_56.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/6f46e39e-685a-4b1a-a5ef-7a10394dd121_isrForCrosstalkSources_1124087_56.stderr parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 12 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 12 has memoization hash 1b9c469c3c6e8afd4934514c1bc339a1 parsl.dataflow.memoization INFO: Task 12 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87aa07600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 13 parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/b0cdd2b5-0061-4554-bb3c-12e251d38941_isrForCrosstalkSources_1123956_62.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/b0cdd2b5-0061-4554-bb3c-12e251d38941_isrForCrosstalkSources_1123956_62.stderr parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 13 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 13 has memoization hash 24d78e4ed3286ee133b13868b72125ee parsl.dataflow.memoization INFO: Task 13 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87aa077e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 14 parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/0940180a-c6f7-49da-b41e-ef73a05b853d_isrForCrosstalkSources_1123958_10.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/0940180a-c6f7-49da-b41e-ef73a05b853d_isrForCrosstalkSources_1123958_10.stderr parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 14 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 14 has memoization hash ae56e4d2252b9d60930b274abc8063cc parsl.dataflow.memoization INFO: Task 14 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87aa079c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 15 parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/5d48e1e2-a9cb-47ec-9223-766e800fd099_isrForCrosstalkSources_1123954_33.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/5d48e1e2-a9cb-47ec-9223-766e800fd099_isrForCrosstalkSources_1123954_33.stderr parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 15 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 15 has memoization hash c4681c80abb188ba517b88b38b2a764a parsl.dataflow.memoization INFO: Task 15 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87aa07560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 16 parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/f11dd8be-a266-4b2b-80c0-c049b8be4078_isrForCrosstalkSources_1123958_45.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/f11dd8be-a266-4b2b-80c0-c049b8be4078_isrForCrosstalkSources_1123958_45.stderr parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 16 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 16 has memoization hash 90525e4ce4fa3788935b5a92857956fb parsl.dataflow.memoization INFO: Task 16 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87aa07a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 17 parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/e337d022-3bc3-4a07-b864-bc966c1be1a1_isrForCrosstalkSources_1124087_51.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/e337d022-3bc3-4a07-b864-bc966c1be1a1_isrForCrosstalkSources_1124087_51.stderr parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 17 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 17 has memoization hash 4f98e39073b9530098834a3f44c3cca6 parsl.dataflow.memoization INFO: Task 17 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87aa07b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 18 parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/02eccca7-c5aa-4cff-a064-5b3124ebae16_isrForCrosstalkSources_1124087_17.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/02eccca7-c5aa-4cff-a064-5b3124ebae16_isrForCrosstalkSources_1124087_17.stderr parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 18 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 18 has memoization hash 1f5af073de4dd956c129c19c723dedf0 parsl.dataflow.memoization INFO: Task 18 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87aa07c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 19 parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/6ad90c32-3471-459f-867a-4502a1c9db02_isrForCrosstalkSources_1124087_5.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/6ad90c32-3471-459f-867a-4502a1c9db02_isrForCrosstalkSources_1124087_5.stderr parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 19 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 19 has memoization hash ebf6315a054042cf1c1838147df76435 parsl.dataflow.memoization INFO: Task 19 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87aa07920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 20 parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/7e12f0e2-fad8-47e5-a5ac-654bd55622ce_isrForCrosstalkSources_1124087_15.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/7e12f0e2-fad8-47e5-a5ac-654bd55622ce_isrForCrosstalkSources_1124087_15.stderr parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 20 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 20 has memoization hash 7da70dd9cd1ead8fdc175f4bfd5f5d4d parsl.dataflow.memoization INFO: Task 20 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87aa07d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 21 parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/f2b42fee-bed3-48d6-a018-551f29c2d92a_isrForCrosstalkSources_1123956_38.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/f2b42fee-bed3-48d6-a018-551f29c2d92a_isrForCrosstalkSources_1123956_38.stderr parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 21 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 21 has memoization hash 93c06abf8b1dfd76e72498f38b420309 parsl.dataflow.memoization INFO: Task 21 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87aa07ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 22 parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/276d2e56-ea67-4dd7-82c5-10fec5f48356_isrForCrosstalkSources_1123954_32.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/276d2e56-ea67-4dd7-82c5-10fec5f48356_isrForCrosstalkSources_1123954_32.stderr parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 22 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash 020020d7740f216ec5823a1af4e5705b parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87aa07e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 23 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/84afb1e2-b8cf-4525-aee9-1df02e39f4ca_isrForCrosstalkSources_1123958_28.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/84afb1e2-b8cf-4525-aee9-1df02e39f4ca_isrForCrosstalkSources_1123958_28.stderr parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 23 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 23 has memoization hash b1f3edf8048b95206b281c1e3eec6754 parsl.dataflow.memoization INFO: Task 23 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87aa07ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 23 try 0 launched on executor multi with executor id 24 parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/c72b5be8-277d-4a53-8f8e-73388aa72d50_isrForCrosstalkSources_1123956_40.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/c72b5be8-277d-4a53-8f8e-73388aa72d50_isrForCrosstalkSources_1123956_40.stderr parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 24 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 24 has memoization hash 7d59264b38bbdea5d6a59e5385a0bbb7 parsl.dataflow.memoization INFO: Task 24 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87aa07ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 25 parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/6fe2aa92-7d72-40a9-9665-45c65f38b8b7_isrForCrosstalkSources_1124087_12.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/6fe2aa92-7d72-40a9-9665-45c65f38b8b7_isrForCrosstalkSources_1124087_12.stderr parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 25 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 25 has memoization hash dbfa4535c55d92836a816b751cad62c0 parsl.dataflow.memoization INFO: Task 25 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f00e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 26 parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/ccf3160c-e231-4dd8-a9b7-a6fbca05db27_isrForCrosstalkSources_1123954_50.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/ccf3160c-e231-4dd8-a9b7-a6fbca05db27_isrForCrosstalkSources_1123954_50.stderr parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 26 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 26 has memoization hash 14eb09a0eb5cbfa0a57691227fcced21 parsl.dataflow.memoization INFO: Task 26 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f0220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 27 parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/575c2233-1f47-4acf-895e-3bad555ec38d_isrForCrosstalkSources_1123954_23.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/575c2233-1f47-4acf-895e-3bad555ec38d_isrForCrosstalkSources_1123954_23.stderr parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 27 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 27 has memoization hash a11b07f5fc390e58ed267053ddc33c84 parsl.dataflow.memoization INFO: Task 27 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f0180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 28 parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/5c10d107-008b-4812-93f0-76e93b2f5d73_isrForCrosstalkSources_1123954_22.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/5c10d107-008b-4812-93f0-76e93b2f5d73_isrForCrosstalkSources_1123954_22.stderr parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 28 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash 19b139e8ecc6fba0d8d07f42e3999359 parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f02c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 29 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/d3c62696-6396-4c76-814d-11785eb5c23c_isrForCrosstalkSources_1123954_16.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/d3c62696-6396-4c76-814d-11785eb5c23c_isrForCrosstalkSources_1123954_16.stderr parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 29 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 29 has memoization hash f18974b03f8436c24ec6d0b462c69fcb parsl.dataflow.memoization INFO: Task 29 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f0360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 29 try 0 launched on executor multi with executor id 30 parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/574a538a-cbdc-46e6-98de-10af6e7194db_isrForCrosstalkSources_1123954_11.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/574a538a-cbdc-46e6-98de-10af6e7194db_isrForCrosstalkSources_1123954_11.stderr parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 30 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 30 has memoization hash 4defc78a8b2866da808743470ab9bd46 parsl.dataflow.memoization INFO: Task 30 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f0400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 31 parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/e971ce1a-a910-421d-940a-f8e2b07912fe_isrForCrosstalkSources_1124087_11.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/e971ce1a-a910-421d-940a-f8e2b07912fe_isrForCrosstalkSources_1124087_11.stderr parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 31 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 31 has memoization hash a4654d4715aea011370094a6a54e8dc8 parsl.dataflow.memoization INFO: Task 31 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 32 parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/80efa6f7-3978-4edd-a9a1-51b099d78d1d_isrForCrosstalkSources_1123958_55.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/80efa6f7-3978-4edd-a9a1-51b099d78d1d_isrForCrosstalkSources_1123958_55.stderr parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 32 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 32 has memoization hash 22628a72f6990044d74b7125057591e5 parsl.dataflow.memoization INFO: Task 32 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f04a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 33 parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/bd4ec73e-2b0e-4423-bbbf-039032933cdd_isrForCrosstalkSources_1123958_34.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/bd4ec73e-2b0e-4423-bbbf-039032933cdd_isrForCrosstalkSources_1123958_34.stderr parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 33 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 33 has memoization hash 7ebdc7d571983ea4af38a15d519059fb parsl.dataflow.memoization INFO: Task 33 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f05e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 34 parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/ef2ab5a8-4ad5-46e3-ad47-e8bef77ed6dc_isrForCrosstalkSources_1123954_6.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/ef2ab5a8-4ad5-46e3-ad47-e8bef77ed6dc_isrForCrosstalkSources_1123954_6.stderr parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 34 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash 231e6d95f629e6ee2c87720495cfd50c parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f0680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 35 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/1f226936-3ce9-4dab-b8cb-d2d677d766d3_isrForCrosstalkSources_1123958_30.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/1f226936-3ce9-4dab-b8cb-d2d677d766d3_isrForCrosstalkSources_1123958_30.stderr parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 35 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 35 has memoization hash 9aaa72de14162f50c8e4729b00165009 parsl.dataflow.memoization INFO: Task 35 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f0720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 35 try 0 launched on executor multi with executor id 36 parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/07c97ef6-8e48-4554-a8b5-b1a3b214fb92_isrForCrosstalkSources_1123958_46.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/07c97ef6-8e48-4554-a8b5-b1a3b214fb92_isrForCrosstalkSources_1123958_46.stderr parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 36 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 36 has memoization hash 227060416eb2d0fc5ff2112335d0c5ad parsl.dataflow.memoization INFO: Task 36 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f0860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 37 parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/e0f29fca-715c-4814-84e2-bfadc5f1d95e_isrForCrosstalkSources_1123956_5.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/e0f29fca-715c-4814-84e2-bfadc5f1d95e_isrForCrosstalkSources_1123956_5.stderr parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 37 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 37 has memoization hash 0480b53c0b389b785cf8fcdd91b6cbf3 parsl.dataflow.memoization INFO: Task 37 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f07c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 38 parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/9a355d03-dd39-4d87-88a8-d07a468bc9d7_isrForCrosstalkSources_1123956_33.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/9a355d03-dd39-4d87-88a8-d07a468bc9d7_isrForCrosstalkSources_1123956_33.stderr parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 38 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 38 has memoization hash a725c27d446ff7837cae0f25c52a83d8 parsl.dataflow.memoization INFO: Task 38 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f0900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 39 parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/c1774e1a-3227-4b20-b606-018197913b07_isrForCrosstalkSources_1124087_33.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/c1774e1a-3227-4b20-b606-018197913b07_isrForCrosstalkSources_1124087_33.stderr parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 39 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 39 has memoization hash 2c33000ac77f1b27d48a4e9b163ce9ef parsl.dataflow.memoization INFO: Task 39 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f09a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 40 parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/0b2216f5-93a9-404f-83af-960f718e0bbf_isrForCrosstalkSources_1123956_11.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/0b2216f5-93a9-404f-83af-960f718e0bbf_isrForCrosstalkSources_1123956_11.stderr parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 40 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 40 has memoization hash 067ac8eeb962dd30ecc75fb8327a3e66 parsl.dataflow.memoization INFO: Task 40 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f0a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 41 parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/eb75578c-d5de-4eba-a593-41aa7b72882d_isrForCrosstalkSources_1123956_29.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/eb75578c-d5de-4eba-a593-41aa7b72882d_isrForCrosstalkSources_1123956_29.stderr parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 41 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 41 has memoization hash 38319fc7aad97bff1601e9592978ceb5 parsl.dataflow.memoization INFO: Task 41 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 42 parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/872f7a5b-4023-47d5-a21c-a9953b878572_isrForCrosstalkSources_1123954_48.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/872f7a5b-4023-47d5-a21c-a9953b878572_isrForCrosstalkSources_1123954_48.stderr parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 42 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 42 has memoization hash 012d356e22ceca5b10d768eeefea7265 parsl.dataflow.memoization INFO: Task 42 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f0ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 43 parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/0ce32f18-e3a4-443b-a02d-94a9aa1c345f_isrForCrosstalkSources_1123956_21.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/0ce32f18-e3a4-443b-a02d-94a9aa1c345f_isrForCrosstalkSources_1123956_21.stderr parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 43 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 43 has memoization hash 2178bb7707aaff08e2e029bd3f3f0197 parsl.dataflow.memoization INFO: Task 43 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f0c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 44 parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/20912cf1-26de-4a61-b947-603f7f179329_isrForCrosstalkSources_1123956_19.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/20912cf1-26de-4a61-b947-603f7f179329_isrForCrosstalkSources_1123956_19.stderr parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 44 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 44 has memoization hash 310dd9e185726bf53f9e8cc0fcc52af4 parsl.dataflow.memoization INFO: Task 44 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f0cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 45 parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/cee6d43b-65ae-40a9-9743-5db753a02cc6_isrForCrosstalkSources_1123954_27.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/cee6d43b-65ae-40a9-9743-5db753a02cc6_isrForCrosstalkSources_1123954_27.stderr parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 45 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 45 has memoization hash 3801b8b790fbfe980663ba56b4fab2b0 parsl.dataflow.memoization INFO: Task 45 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f0d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 46 parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/95e69fd7-6fe5-4de6-9fae-acabd1eb86ed_isrForCrosstalkSources_1124087_54.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/95e69fd7-6fe5-4de6-9fae-acabd1eb86ed_isrForCrosstalkSources_1124087_54.stderr parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 46 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 46 has memoization hash 7b4e4af1d77c797fe9f0c965de258100 parsl.dataflow.memoization INFO: Task 46 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 47 parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/506bba7e-88df-4fb3-bdfc-4abb29669499_isrForCrosstalkSources_1123956_15.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/506bba7e-88df-4fb3-bdfc-4abb29669499_isrForCrosstalkSources_1123956_15.stderr parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 47 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 47 has memoization hash 32ad066e1da9b954bc67a0d0f3c9326d parsl.dataflow.memoization INFO: Task 47 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f0e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 48 parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/0fbc9a99-03c9-44fd-8a86-7f4a67630c5b_isrForCrosstalkSources_1123958_17.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/0fbc9a99-03c9-44fd-8a86-7f4a67630c5b_isrForCrosstalkSources_1123958_17.stderr parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 48 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 48 has memoization hash 7c8ec3a8094e62d1fc10d380838e8548 parsl.dataflow.memoization INFO: Task 48 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f0f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 49 parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/0cc450a1-4c04-4953-9b72-f1b6e2a5cfae_isrForCrosstalkSources_1123956_7.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/0cc450a1-4c04-4953-9b72-f1b6e2a5cfae_isrForCrosstalkSources_1123956_7.stderr parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 49 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 49 has memoization hash 7dfc33d7d1b93df3648c565403cb04e3 parsl.dataflow.memoization INFO: Task 49 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f0fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 50 parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/275d9377-c990-4aba-9d0f-4c5e3c686273_isrForCrosstalkSources_1123954_47.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/275d9377-c990-4aba-9d0f-4c5e3c686273_isrForCrosstalkSources_1123954_47.stderr parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 50 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 50 has memoization hash adf2795b707fe4f0c8149c416a31ea1c parsl.dataflow.memoization INFO: Task 50 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 51 parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/4f4208b7-fc72-4cc3-ba61-f5735ff24222_isrForCrosstalkSources_1123958_26.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/4f4208b7-fc72-4cc3-ba61-f5735ff24222_isrForCrosstalkSources_1123958_26.stderr parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 51 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 51 has memoization hash d144575f2ad57d23e1a5bdc688a3248c parsl.dataflow.memoization INFO: Task 51 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 52 parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/f173d899-4afa-40f8-8afd-1e36a4784455_isrForCrosstalkSources_1123958_50.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/f173d899-4afa-40f8-8afd-1e36a4784455_isrForCrosstalkSources_1123958_50.stderr parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 52 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash ec360ba692371a2e80cfd1f8d949c526 parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f1120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 53 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/576bbdae-d433-4f36-bad3-e66a577e288b_isrForCrosstalkSources_1124087_6.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/576bbdae-d433-4f36-bad3-e66a577e288b_isrForCrosstalkSources_1124087_6.stderr parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 53 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 53 has memoization hash 4b6403c65aacc6a75007b7f361da5de2 parsl.dataflow.memoization INFO: Task 53 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f1260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 53 try 0 launched on executor multi with executor id 54 parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/1b50ad35-57fc-4140-a7bc-35209ba4f44b_isrForCrosstalkSources_1124087_44.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/1b50ad35-57fc-4140-a7bc-35209ba4f44b_isrForCrosstalkSources_1124087_44.stderr parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 54 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 54 has memoization hash 9c4cd2becef2b60a85e8f235cfd15680 parsl.dataflow.memoization INFO: Task 54 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f1300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 55 parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/bc1d1bf8-d843-40ef-bf68-135ca854f8ab_isrForCrosstalkSources_1123958_1.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/bc1d1bf8-d843-40ef-bf68-135ca854f8ab_isrForCrosstalkSources_1123958_1.stderr parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 55 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 55 has memoization hash 3d5993f0e746c523fd74a2a78cbc081c parsl.dataflow.memoization INFO: Task 55 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f13a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 56 parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/aee6b67b-c26d-42a8-bef5-4c24e043e4ec_isrForCrosstalkSources_1123954_43.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/aee6b67b-c26d-42a8-bef5-4c24e043e4ec_isrForCrosstalkSources_1123954_43.stderr parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 56 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 56 has memoization hash b5f5f2bef0b0efa432a96f605d41642b parsl.dataflow.memoization INFO: Task 56 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f14e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 57 parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/9fac5b87-9c0a-46c3-a2c4-669ae8b9c81f_isrForCrosstalkSources_1123954_3.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/9fac5b87-9c0a-46c3-a2c4-669ae8b9c81f_isrForCrosstalkSources_1123954_3.stderr parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 57 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 57 has memoization hash 2b370d1356913ca021fa0237f5909fca parsl.dataflow.memoization INFO: Task 57 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f1440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 58 parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/0d1a2d9f-e8e9-4684-bee5-05bb484cda3d_isrForCrosstalkSources_1123954_14.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/0d1a2d9f-e8e9-4684-bee5-05bb484cda3d_isrForCrosstalkSources_1123954_14.stderr parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 58 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 58 has memoization hash 606a9a43fa8b6f298d4707551f339a11 parsl.dataflow.memoization INFO: Task 58 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f1580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 59 parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/58622b96-908f-49bd-a6f2-fd25ae00cc6d_isrForCrosstalkSources_1123956_30.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/58622b96-908f-49bd-a6f2-fd25ae00cc6d_isrForCrosstalkSources_1123956_30.stderr parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 59 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 59 has memoization hash ac254384f21b3b2055417a87155a722f parsl.dataflow.memoization INFO: Task 59 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f1620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 60 parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/59b74ac7-08ce-4e6e-8186-a3ef96d91ee7_isrForCrosstalkSources_1123956_6.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/59b74ac7-08ce-4e6e-8186-a3ef96d91ee7_isrForCrosstalkSources_1123956_6.stderr parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 60 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 60 has memoization hash fa0ac73e6a99c0df5156ee8a15dfbd63 parsl.dataflow.memoization INFO: Task 60 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 61 parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/0bc8f37e-47f8-47ca-ac39-bd68f3d7621a_isrForCrosstalkSources_1123956_13.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/0bc8f37e-47f8-47ca-ac39-bd68f3d7621a_isrForCrosstalkSources_1123956_13.stderr parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 61 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 61 has memoization hash 4d6d4cda71002eae7bbf1e5623a1f0fb parsl.dataflow.memoization INFO: Task 61 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 62 parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/b2defb6e-d680-4b67-86d7-17e91d2cc90b_isrForCrosstalkSources_1123956_45.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/b2defb6e-d680-4b67-86d7-17e91d2cc90b_isrForCrosstalkSources_1123956_45.stderr parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 62 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 62 has memoization hash e2408ada73906cbb78a2ccec418ec5fb parsl.dataflow.memoization INFO: Task 62 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f1760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 63 parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/6515e6a8-194a-44cd-8306-a74d82222bed_isrForCrosstalkSources_1123958_56.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/6515e6a8-194a-44cd-8306-a74d82222bed_isrForCrosstalkSources_1123958_56.stderr parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 63 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 63 has memoization hash e4ef4f5fc433952a20bd37ce82003ad7 parsl.dataflow.memoization INFO: Task 63 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f18a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 64 parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/e103b8ad-111a-4b1c-b1ad-6135abf9f959_isrForCrosstalkSources_1123958_25.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/e103b8ad-111a-4b1c-b1ad-6135abf9f959_isrForCrosstalkSources_1123958_25.stderr parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 64 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 64 has memoization hash bb91a40120038786c0bbe0861c066ca5 parsl.dataflow.memoization INFO: Task 64 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f1940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 65 parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/0b5ca7dc-7dcf-466d-af7a-a2f0bcdb7f0c_isrForCrosstalkSources_1123956_25.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/0b5ca7dc-7dcf-466d-af7a-a2f0bcdb7f0c_isrForCrosstalkSources_1123956_25.stderr parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 65 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 65 has memoization hash 3cd4ec5d53756be22f40eee186e2595f parsl.dataflow.memoization INFO: Task 65 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 65 try 0 launched on executor multi with executor id 66 parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/a8c169bc-ff2c-416c-be79-9fdeb5f3f8f2_isrForCrosstalkSources_1123958_23.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/a8c169bc-ff2c-416c-be79-9fdeb5f3f8f2_isrForCrosstalkSources_1123958_23.stderr parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 66 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 66 has memoization hash c3775abce6c1fe606140894c6dbcaf41 parsl.dataflow.memoization INFO: Task 66 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 67 parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/ea21f981-0fb2-4527-9c9c-a2f39fb3ef0d_isrForCrosstalkSources_1123956_44.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/ea21f981-0fb2-4527-9c9c-a2f39fb3ef0d_isrForCrosstalkSources_1123956_44.stderr parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 67 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 67 has memoization hash df642638f72b36d57b87ad1a321ede9d parsl.dataflow.memoization INFO: Task 67 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f1a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 68 parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/b5794b80-9059-4385-977e-aec0d132e02c_isrForCrosstalkSources_1123954_39.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/b5794b80-9059-4385-977e-aec0d132e02c_isrForCrosstalkSources_1123954_39.stderr parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 68 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 68 has memoization hash af61fac49737e1910d04fd2b20a08e5b parsl.dataflow.memoization INFO: Task 68 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f1bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 69 parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/937a37e1-cd9e-4458-b333-4aea36e6ed9d_isrForCrosstalkSources_1124087_62.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/937a37e1-cd9e-4458-b333-4aea36e6ed9d_isrForCrosstalkSources_1124087_62.stderr parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 69 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 69 has memoization hash da6614574f32209a8b9e158a3d08e88c parsl.dataflow.memoization INFO: Task 69 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f1c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 70 parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/76ac6ef4-2f69-446d-bde8-fb6b9ef838fb_isrForCrosstalkSources_1123956_54.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/76ac6ef4-2f69-446d-bde8-fb6b9ef838fb_isrForCrosstalkSources_1123956_54.stderr parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 70 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 70 has memoization hash 2be97232539d66c9029570f44f769428 parsl.dataflow.memoization INFO: Task 70 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 71 parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/291d2d0e-ae5b-425e-95f4-acd7833cad8b_isrForCrosstalkSources_1123954_34.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/291d2d0e-ae5b-425e-95f4-acd7833cad8b_isrForCrosstalkSources_1123954_34.stderr parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 71 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 71 has memoization hash d351488a0cddacd1f5de9027417f4404 parsl.dataflow.memoization INFO: Task 71 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f1e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 71 try 0 launched on executor multi with executor id 72 parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/fbc4616f-1322-46f5-914a-a140b66a14a1_isrForCrosstalkSources_1124087_28.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/fbc4616f-1322-46f5-914a-a140b66a14a1_isrForCrosstalkSources_1124087_28.stderr parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 72 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 72 has memoization hash 645b0960a80e89c5f8cac409179c0b6b parsl.dataflow.memoization INFO: Task 72 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f1da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 73 parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/42543483-f2d0-46f1-bc7f-09b0006f1ef0_isrForCrosstalkSources_1123958_48.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/42543483-f2d0-46f1-bc7f-09b0006f1ef0_isrForCrosstalkSources_1123958_48.stderr parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 73 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 73 has memoization hash 811d5c9297d1ef2daa1e66504f708ac3 parsl.dataflow.memoization INFO: Task 73 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f1ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 74 parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/06000f64-d9f4-4b27-b1d9-eb2b79373752_isrForCrosstalkSources_1123954_24.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/06000f64-d9f4-4b27-b1d9-eb2b79373752_isrForCrosstalkSources_1123954_24.stderr parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 74 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 74 has memoization hash c187c9a79cb2f99d3c22a13c9fc1b3a0 parsl.dataflow.memoization INFO: Task 74 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f1f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 75 parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/aa83b164-f09d-404c-ace9-43d5fc9f6001_isrForCrosstalkSources_1123956_37.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/aa83b164-f09d-404c-ace9-43d5fc9f6001_isrForCrosstalkSources_1123956_37.stderr parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 75 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 75 has memoization hash 327b536576bbac117c7fb5a9b3367afb parsl.dataflow.memoization INFO: Task 75 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f2020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 76 parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/70a8a9b0-e325-4de5-b36a-fa4287286950_isrForCrosstalkSources_1123958_12.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/70a8a9b0-e325-4de5-b36a-fa4287286950_isrForCrosstalkSources_1123958_12.stderr parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 76 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash aa7b93a5a0fa4f531b2c9fd6616cc231 parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 77 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/9dfb1ae0-4c55-4c36-a00b-fceb83a72ebe_isrForCrosstalkSources_1123954_41.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/9dfb1ae0-4c55-4c36-a00b-fceb83a72ebe_isrForCrosstalkSources_1123954_41.stderr parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 77 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 77 has memoization hash ad0ddf5872a46314d93055f4e171ab19 parsl.dataflow.memoization INFO: Task 77 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f20c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 77 try 0 launched on executor multi with executor id 78 parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/e87e6255-4323-48ce-abfa-ec6480abf5ee_isrForCrosstalkSources_1124087_47.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/e87e6255-4323-48ce-abfa-ec6480abf5ee_isrForCrosstalkSources_1124087_47.stderr parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 78 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 78 has memoization hash da3e8badf98c890544389a279838a98e parsl.dataflow.memoization INFO: Task 78 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f2200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 79 parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/a0d52380-1bfd-4e26-97d1-dff0ec256b42_isrForCrosstalkSources_1124087_43.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/a0d52380-1bfd-4e26-97d1-dff0ec256b42_isrForCrosstalkSources_1124087_43.stderr parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 79 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 79 has memoization hash 7e59e7be6885c2d518184fd56bd032a2 parsl.dataflow.memoization INFO: Task 79 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f22a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 80 parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/fac16bcc-dd5e-442a-b61f-5ca8d78cfd09_isrForCrosstalkSources_1123956_23.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/fac16bcc-dd5e-442a-b61f-5ca8d78cfd09_isrForCrosstalkSources_1123956_23.stderr parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 80 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 80 has memoization hash c322ccb014805c24eb05724adffb3872 parsl.dataflow.memoization INFO: Task 80 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f2340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 81 parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/f8796ce1-a33f-450e-8274-17c21ba4c9ef_isrForCrosstalkSources_1124087_4.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/f8796ce1-a33f-450e-8274-17c21ba4c9ef_isrForCrosstalkSources_1124087_4.stderr parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 81 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 81 has memoization hash 6842875bacc61b41064ef481781d1464 parsl.dataflow.memoization INFO: Task 81 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f2480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 82 parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/9dd5ccc0-fa0a-43fa-8376-ac56c34feaab_isrForCrosstalkSources_1123958_9.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/9dd5ccc0-fa0a-43fa-8376-ac56c34feaab_isrForCrosstalkSources_1123958_9.stderr parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 82 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 82 has memoization hash f723c95841942f8a1341beab91c533e9 parsl.dataflow.memoization INFO: Task 82 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f23e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 83 parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/fcba0bde-fc56-463f-a71a-af7540ee366c_isrForCrosstalkSources_1123958_22.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/fcba0bde-fc56-463f-a71a-af7540ee366c_isrForCrosstalkSources_1123958_22.stderr parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 83 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 83 has memoization hash 830965738b067cef64dcb2844f615cd9 parsl.dataflow.memoization INFO: Task 83 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f2520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 83 try 0 launched on executor multi with executor id 84 parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/bbbc92ed-cfa8-47a8-887e-ee2fde097336_isrForCrosstalkSources_1123954_51.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/bbbc92ed-cfa8-47a8-887e-ee2fde097336_isrForCrosstalkSources_1123954_51.stderr parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 84 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 84 has memoization hash e0f3e7a421edd789af94364bcc575599 parsl.dataflow.memoization INFO: Task 84 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f25c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 85 parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/3cb27e9f-30a8-472a-a971-55594cb6d546_isrForCrosstalkSources_1123954_12.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/3cb27e9f-30a8-472a-a971-55594cb6d546_isrForCrosstalkSources_1123954_12.stderr parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 85 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 85 has memoization hash cc504ef86d000e012cadc9889b8dee17 parsl.dataflow.memoization INFO: Task 85 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 86 parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/3f60ea24-28a8-40c5-ac2e-3529594d7e6c_isrForCrosstalkSources_1123954_62.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/3f60ea24-28a8-40c5-ac2e-3529594d7e6c_isrForCrosstalkSources_1123954_62.stderr parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 86 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 86 has memoization hash 59410c807a22c55a476e935caa36e33a parsl.dataflow.memoization INFO: Task 86 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 87 parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/6bf6a007-adfb-4f29-b546-33b51ef1ef44_isrForCrosstalkSources_1123954_5.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/6bf6a007-adfb-4f29-b546-33b51ef1ef44_isrForCrosstalkSources_1123954_5.stderr parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 87 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 87 has memoization hash 1af97dc7391513a04cae702888d87a63 parsl.dataflow.memoization INFO: Task 87 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f2700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 88 parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/453aac3f-0299-4253-af66-92ac6b3d3bff_isrForCrosstalkSources_1123958_7.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/453aac3f-0299-4253-af66-92ac6b3d3bff_isrForCrosstalkSources_1123958_7.stderr parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 88 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 88 has memoization hash fa1302db87ec6f18942e934d7d70366e parsl.dataflow.memoization INFO: Task 88 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f2840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 89 parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/80ba5282-8dfc-4970-9904-3a9c49229f09_isrForCrosstalkSources_1123958_11.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/80ba5282-8dfc-4970-9904-3a9c49229f09_isrForCrosstalkSources_1123958_11.stderr parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 89 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 89 has memoization hash 3263f19f18bfec32dffd44a8fb7f58e0 parsl.dataflow.memoization INFO: Task 89 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f28e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 90 parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/3c56078a-bb66-4d56-bdae-a48975735bea_isrForCrosstalkSources_1124087_16.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/3c56078a-bb66-4d56-bdae-a48975735bea_isrForCrosstalkSources_1124087_16.stderr parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 90 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 90 has memoization hash dd7cd990a8f49d5a1e9dade52be736e5 parsl.dataflow.memoization INFO: Task 90 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 91 parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/4549f735-cb71-41c9-8259-cb5bc3355852_isrForCrosstalkSources_1123954_19.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/4549f735-cb71-41c9-8259-cb5bc3355852_isrForCrosstalkSources_1123954_19.stderr parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 91 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 91 has memoization hash 5a91210bfe8357ef3167b75a156a0784 parsl.dataflow.memoization INFO: Task 91 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f2ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 91 try 0 launched on executor multi with executor id 92 parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/2988226c-ec86-4762-acc6-acab7176638d_isrForCrosstalkSources_1123954_10.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/2988226c-ec86-4762-acc6-acab7176638d_isrForCrosstalkSources_1123954_10.stderr parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 92 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 92 has memoization hash fafa903d4946adf4801e92a3641ce34c parsl.dataflow.memoization INFO: Task 92 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f2a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 93 parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/d660f5a9-e85f-4353-bbb1-ebe62dfe9dd9_isrForCrosstalkSources_1123956_20.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/d660f5a9-e85f-4353-bbb1-ebe62dfe9dd9_isrForCrosstalkSources_1123956_20.stderr parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 93 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 93 has memoization hash 48d991c19eec7f0bea16af7623b90330 parsl.dataflow.memoization INFO: Task 93 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 94 parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/98fa5e2a-f825-4c7c-9e28-b8782e9670a6_isrForCrosstalkSources_1124087_7.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/98fa5e2a-f825-4c7c-9e28-b8782e9670a6_isrForCrosstalkSources_1124087_7.stderr parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 94 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 94 has memoization hash 7f2c4b8c0e90bfacc0b9ffe36c76751d parsl.dataflow.memoization INFO: Task 94 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f2c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 95 parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/08b752c2-c33f-4b4b-b4e0-e83ae7f2164d_isrForCrosstalkSources_1123954_20.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/08b752c2-c33f-4b4b-b4e0-e83ae7f2164d_isrForCrosstalkSources_1123954_20.stderr parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 95 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 95 has memoization hash 63f97caf01e8cca628f2b30b6ac18d68 parsl.dataflow.memoization INFO: Task 95 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f2ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 96 parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/7082a77c-9ef4-423d-95a2-2b96853150ad_isrForCrosstalkSources_1123954_37.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/7082a77c-9ef4-423d-95a2-2b96853150ad_isrForCrosstalkSources_1123954_37.stderr parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 96 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 96 has memoization hash 0d33ea46c0d61979cd4e12fea2a155dc parsl.dataflow.memoization INFO: Task 96 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 97 parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/ac707209-f50f-4d9e-b9b9-5b9d12621d95_isrForCrosstalkSources_1124087_14.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/ac707209-f50f-4d9e-b9b9-5b9d12621d95_isrForCrosstalkSources_1124087_14.stderr parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 97 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 97 has memoization hash 9b64abcdfeefd28a8d784a869948068b parsl.dataflow.memoization INFO: Task 97 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f2d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 98 parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/59cc7fd9-0794-4683-8f4d-1c7d31af6a83_isrForCrosstalkSources_1124087_46.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/59cc7fd9-0794-4683-8f4d-1c7d31af6a83_isrForCrosstalkSources_1124087_46.stderr parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 98 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 98 has memoization hash bc500fa263102f0f952d6397368bdd9e parsl.dataflow.memoization INFO: Task 98 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f2e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 99 parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/a0630e0a-e076-493a-8729-890d1fa73cbc_isrForCrosstalkSources_1123954_38.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/a0630e0a-e076-493a-8729-890d1fa73cbc_isrForCrosstalkSources_1123954_38.stderr parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 99 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 99 has memoization hash 65b43fb1c2908f0ee0d7fd82545417d7 parsl.dataflow.memoization INFO: Task 99 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 100 parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/1ac3a382-b337-450a-be30-afa68a0ff057_isrForCrosstalkSources_1123954_42.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/1ac3a382-b337-450a-be30-afa68a0ff057_isrForCrosstalkSources_1123954_42.stderr parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 100 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash 49d8ba424cd6467b0cc1c9e2516b258f parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f2fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 101 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/676d10d8-ef04-4ce0-b7a9-234df7eb8592_isrForCrosstalkSources_1124087_30.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/676d10d8-ef04-4ce0-b7a9-234df7eb8592_isrForCrosstalkSources_1124087_30.stderr parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 101 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 101 has memoization hash 72d6f68fe5491e5856181da6772a7be7 parsl.dataflow.memoization INFO: Task 101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f3100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 101 try 0 launched on executor multi with executor id 102 parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/d923848f-e9b4-4669-8a86-d68491eff739_isrForCrosstalkSources_1123958_5.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/d923848f-e9b4-4669-8a86-d68491eff739_isrForCrosstalkSources_1123958_5.stderr parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 102 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 102 has memoization hash 4bc824b6de1c2200872cb4715f5a9de1 parsl.dataflow.memoization INFO: Task 102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f3060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 103 parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/3aae3930-48c0-4c52-952b-c4cf48684818_isrForCrosstalkSources_1123954_9.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/3aae3930-48c0-4c52-952b-c4cf48684818_isrForCrosstalkSources_1123954_9.stderr parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 103 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 103 has memoization hash 88491b14893bbceeff929a9158c05af4 parsl.dataflow.memoization INFO: Task 103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f31a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 104 parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/69f3c2b9-9b80-405c-bd26-eedd87ad7bc0_isrForCrosstalkSources_1124087_40.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/69f3c2b9-9b80-405c-bd26-eedd87ad7bc0_isrForCrosstalkSources_1124087_40.stderr parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 104 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 104 has memoization hash 1a5ba1773ec2d52827d9f83f8c3dbf24 parsl.dataflow.memoization INFO: Task 104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f3240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 105 parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/a39b13f8-51f5-465c-be1d-abde392b9fb6_isrForCrosstalkSources_1123954_55.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/a39b13f8-51f5-465c-be1d-abde392b9fb6_isrForCrosstalkSources_1123954_55.stderr parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 105 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 105 has memoization hash 9b7a9681aa47321d65b664c4d53f8eb8 parsl.dataflow.memoization INFO: Task 105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f32e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 106 parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/2d4dec71-b517-4130-a0d0-638f24458f36_isrForCrosstalkSources_1123958_53.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/2d4dec71-b517-4130-a0d0-638f24458f36_isrForCrosstalkSources_1123958_53.stderr parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 106 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 106 has memoization hash 39c955f9482c5dbcdab9489a83159520 parsl.dataflow.memoization INFO: Task 106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 107 parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/fb496018-2590-4a36-b5ba-34beb55528d5_isrForCrosstalkSources_1123954_46.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/fb496018-2590-4a36-b5ba-34beb55528d5_isrForCrosstalkSources_1123954_46.stderr parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 107 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 107 has memoization hash d705b5e1a45d887c0f7e503cdc406af9 parsl.dataflow.memoization INFO: Task 107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f3380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 108 parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/d720f84f-c9c1-4437-ae48-c9af7d7b62da_isrForCrosstalkSources_1124087_32.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/d720f84f-c9c1-4437-ae48-c9af7d7b62da_isrForCrosstalkSources_1124087_32.stderr parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 108 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 108 has memoization hash b995076345343a245aebb98b128b69e0 parsl.dataflow.memoization INFO: Task 108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f34c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 109 parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/37ff3efe-5e02-4569-a24f-c94b8ead2772_isrForCrosstalkSources_1124087_1.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/37ff3efe-5e02-4569-a24f-c94b8ead2772_isrForCrosstalkSources_1124087_1.stderr parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 109 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 109 has memoization hash 9c8c9f0a5b2c3fbb085c3b8524a87aeb parsl.dataflow.memoization INFO: Task 109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f3560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 110 parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/dfe134bd-4d3f-4922-937a-009cc0a9b783_isrForCrosstalkSources_1123954_21.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/dfe134bd-4d3f-4922-937a-009cc0a9b783_isrForCrosstalkSources_1123954_21.stderr parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 110 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 110 has memoization hash f4277506166d5fde3b5a6433df536cd6 parsl.dataflow.memoization INFO: Task 110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f3600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 111 parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/14966996-bc37-467d-bccf-66c6127e28de_isrForCrosstalkSources_1124087_29.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/14966996-bc37-467d-bccf-66c6127e28de_isrForCrosstalkSources_1124087_29.stderr parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 111 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 111 has memoization hash aa7f5a2911b0642677b4f9f2947bc67d parsl.dataflow.memoization INFO: Task 111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f3740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 112 parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/0ad32afc-f447-443f-a6bc-946d99e410d3_isrForCrosstalkSources_1123956_58.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/0ad32afc-f447-443f-a6bc-946d99e410d3_isrForCrosstalkSources_1123956_58.stderr parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 112 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 112 has memoization hash d89e4bdd3837f908833a80f3b2f0035f parsl.dataflow.memoization INFO: Task 112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f36a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 113 parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/872fafd6-890f-43b7-9591-a9de07a728f6_isrForCrosstalkSources_1123956_43.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/872fafd6-890f-43b7-9591-a9de07a728f6_isrForCrosstalkSources_1123956_43.stderr parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 113 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 113 has memoization hash ca95c99f264313141a07e9c983722c84 parsl.dataflow.memoization INFO: Task 113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f37e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 114 parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/54f7a794-a96b-4260-b984-65cb9bd6f1b9_isrForCrosstalkSources_1124087_22.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/54f7a794-a96b-4260-b984-65cb9bd6f1b9_isrForCrosstalkSources_1124087_22.stderr parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 114 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 114 has memoization hash 01f0499161e0d9b68291f6be8aa92835 parsl.dataflow.memoization INFO: Task 114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f3880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 115 parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/4f46ce8c-b0d7-450e-9a35-62e9d291ee4e_isrForCrosstalkSources_1123958_49.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/4f46ce8c-b0d7-450e-9a35-62e9d291ee4e_isrForCrosstalkSources_1123958_49.stderr parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 115 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 115 has memoization hash 0685316270a68adccacdd7bf233cc848 parsl.dataflow.memoization INFO: Task 115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f3920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 116 parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/89d90e91-cf9c-4e2d-aa52-22dd85df9547_isrForCrosstalkSources_1123956_2.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/89d90e91-cf9c-4e2d-aa52-22dd85df9547_isrForCrosstalkSources_1123956_2.stderr parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 116 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 116 has memoization hash e4bf1a51bbc21ce08c4ac283134498b9 parsl.dataflow.memoization INFO: Task 116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f3a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 117 parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/d9a4ffa4-8714-48e9-9a97-aa37e412b757_isrForCrosstalkSources_1123956_34.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/d9a4ffa4-8714-48e9-9a97-aa37e412b757_isrForCrosstalkSources_1123956_34.stderr parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 117 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 117 has memoization hash 71a064758331336aef566f03765d03a2 parsl.dataflow.memoization INFO: Task 117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f39c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 118 parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/f87f47fa-b8f7-4ff4-99f1-3c31502c62fa_isrForCrosstalkSources_1123958_59.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/f87f47fa-b8f7-4ff4-99f1-3c31502c62fa_isrForCrosstalkSources_1123958_59.stderr parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 118 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash c3f8fb1ea2ce449f791975dbb09fbcf1 parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f3b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 119 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/aa729c4d-f4c4-4192-b0d0-382188322e22_isrForCrosstalkSources_1123958_54.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/aa729c4d-f4c4-4192-b0d0-382188322e22_isrForCrosstalkSources_1123958_54.stderr parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 119 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 119 has memoization hash 3df9830e35f6b184f3761453684fe653 parsl.dataflow.memoization INFO: Task 119 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f3ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 119 try 0 launched on executor multi with executor id 120 parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/7ef0ee61-f593-4a64-bf66-682c439f279e_isrForCrosstalkSources_1123954_31.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/7ef0ee61-f593-4a64-bf66-682c439f279e_isrForCrosstalkSources_1123954_31.stderr parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 120 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 120 has memoization hash 854aa1d50e3242caf0895d2c69813ce6 parsl.dataflow.memoization INFO: Task 120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f3c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 121 parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/ed9fe296-3ee6-42cc-8c2a-2246a1fa8645_isrForCrosstalkSources_1124087_57.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/ed9fe296-3ee6-42cc-8c2a-2246a1fa8645_isrForCrosstalkSources_1124087_57.stderr parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 121 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 121 has memoization hash 927291f6b7fdc49110a550d433280fca parsl.dataflow.memoization INFO: Task 121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f3d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 122 parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/793ffd1d-8f22-4a2a-bc0f-3df2de9af972_isrForCrosstalkSources_1123958_8.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/793ffd1d-8f22-4a2a-bc0f-3df2de9af972_isrForCrosstalkSources_1123958_8.stderr parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 122 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 122 has memoization hash c8e75b4b535dc28c4b43cc5aa8d3de93 parsl.dataflow.memoization INFO: Task 122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f3ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 123 parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/ea9e6cf1-594e-489c-b122-34d1bad1e7c7_isrForCrosstalkSources_1123956_56.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/ea9e6cf1-594e-489c-b122-34d1bad1e7c7_isrForCrosstalkSources_1123956_56.stderr parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 123 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 123 has memoization hash 96c46b39b4d58ced056ba53a3886f1ea parsl.dataflow.memoization INFO: Task 123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 124 parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/b91f811c-50c3-4cf7-9531-52b55cfcc502_isrForCrosstalkSources_1123954_30.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/b91f811c-50c3-4cf7-9531-52b55cfcc502_isrForCrosstalkSources_1123954_30.stderr parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 124 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 124 has memoization hash ec53670fe5b2426b755f3bc208079c83 parsl.dataflow.memoization INFO: Task 124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f3ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 125 parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/54f56583-d516-4d3f-a222-1e421ea648db_isrForCrosstalkSources_1123954_54.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/54f56583-d516-4d3f-a222-1e421ea648db_isrForCrosstalkSources_1123954_54.stderr parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 125 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 125 has memoization hash 08c5f7288f38035349cdb29a01aa2518 parsl.dataflow.memoization INFO: Task 125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 125 try 0 launched on executor multi with executor id 126 parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/227bc79f-5f0f-46d5-83cc-a6ff837eee3c_isrForCrosstalkSources_1123954_2.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/227bc79f-5f0f-46d5-83cc-a6ff837eee3c_isrForCrosstalkSources_1123954_2.stderr parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 126 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 126 has memoization hash a557a69cdd2a4d83f4569beac843acfa parsl.dataflow.memoization INFO: Task 126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a4f0040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 127 parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/d3326318-82c6-4356-8b17-c4546fea4009_isrForCrosstalkSources_1123956_24.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/d3326318-82c6-4356-8b17-c4546fea4009_isrForCrosstalkSources_1123956_24.stderr parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 127 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 127 has memoization hash aa1d3a268c02bedb9a22bda07663e77c parsl.dataflow.memoization INFO: Task 127 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a568040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 127 try 0 launched on executor multi with executor id 128 parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/1e94c2b4-71bb-4d3f-98ef-ebcfb0ec4eef_isrForCrosstalkSources_1123956_39.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/1e94c2b4-71bb-4d3f-98ef-ebcfb0ec4eef_isrForCrosstalkSources_1123956_39.stderr parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 128 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 128 has memoization hash cc81911324debd4921ad034839333a24 parsl.dataflow.memoization INFO: Task 128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a568180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 129 parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/54eaecb4-80e5-4048-87d0-6b314831129c_isrForCrosstalkSources_1123954_60.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/54eaecb4-80e5-4048-87d0-6b314831129c_isrForCrosstalkSources_1123954_60.stderr parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 129 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 129 has memoization hash 1a35ecceea021a536430b90579a1fa65 parsl.dataflow.memoization INFO: Task 129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a568220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 129 try 0 launched on executor multi with executor id 130 parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/b1d4f3c9-db0e-425d-97b1-031c9cbfd1de_isrForCrosstalkSources_1124087_20.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/b1d4f3c9-db0e-425d-97b1-031c9cbfd1de_isrForCrosstalkSources_1124087_20.stderr parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 130 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 130 has memoization hash 61e2e79cf21c8d3b2dc8ac40ffbbb408 parsl.dataflow.memoization INFO: Task 130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a5682c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 131 parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/d35e5ca1-a1ae-4e69-aeb1-84891ae4bb75_isrForCrosstalkSources_1123954_61.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/d35e5ca1-a1ae-4e69-aeb1-84891ae4bb75_isrForCrosstalkSources_1123954_61.stderr parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 131 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 131 has memoization hash 1c92f5905a721f3ac17c1391c97d51b4 parsl.dataflow.memoization INFO: Task 131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a568400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 131 try 0 launched on executor multi with executor id 132 parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/6d7f8559-b8e6-47ef-8c66-9a625019b568_isrForCrosstalkSources_1124087_48.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/6d7f8559-b8e6-47ef-8c66-9a625019b568_isrForCrosstalkSources_1124087_48.stderr parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 132 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 132 has memoization hash 51699c7daa31e33728818cd36acf8000 parsl.dataflow.memoization INFO: Task 132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a568360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 133 parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/3cbb1ba1-c09a-4f8a-bef2-e39b3a35d2a7_isrForCrosstalkSources_1124087_18.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/3cbb1ba1-c09a-4f8a-bef2-e39b3a35d2a7_isrForCrosstalkSources_1124087_18.stderr parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 133 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 133 has memoization hash f373b0cab3b71ac394c8adeb97d275f5 parsl.dataflow.memoization INFO: Task 133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a5684a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 133 try 0 launched on executor multi with executor id 134 parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/57ebdb48-8f3c-4ca2-8fb0-da0f71c195e4_isrForCrosstalkSources_1123958_2.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/57ebdb48-8f3c-4ca2-8fb0-da0f71c195e4_isrForCrosstalkSources_1123958_2.stderr parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 134 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 134 has memoization hash dd39e71bdb3943502b785c2a4b163eb7 parsl.dataflow.memoization INFO: Task 134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a568540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 135 parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/44acff89-0507-4c2f-88db-1eddd0f46f85_isrForCrosstalkSources_1123958_20.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/44acff89-0507-4c2f-88db-1eddd0f46f85_isrForCrosstalkSources_1123958_20.stderr parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 135 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 135 has memoization hash 301cb286aefb4baf6e6f0396f87ef9a9 parsl.dataflow.memoization INFO: Task 135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a5685e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 135 try 0 launched on executor multi with executor id 136 parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/ce701303-d53c-43d5-a717-a61e9449da37_isrForCrosstalkSources_1123956_53.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/ce701303-d53c-43d5-a717-a61e9449da37_isrForCrosstalkSources_1123956_53.stderr parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 136 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 136 has memoization hash b0eaefa6f45893698ad0b8a3dbce06af parsl.dataflow.memoization INFO: Task 136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a568720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 137 parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/56444a86-f1bb-49f7-b5a7-52bc0bea8f47_isrForCrosstalkSources_1124087_59.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/56444a86-f1bb-49f7-b5a7-52bc0bea8f47_isrForCrosstalkSources_1124087_59.stderr parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 137 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 137 has memoization hash affa0b18207738fc6e1fa324bff82bbf parsl.dataflow.memoization INFO: Task 137 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a568680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 137 try 0 launched on executor multi with executor id 138 parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/88566225-38b2-4096-825f-b65e7ca1ae45_isrForCrosstalkSources_1123956_17.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/88566225-38b2-4096-825f-b65e7ca1ae45_isrForCrosstalkSources_1123956_17.stderr parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 138 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 138 has memoization hash 6bc1d32c0151d8b92f8420ad87b7a471 parsl.dataflow.memoization INFO: Task 138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a5687c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 139 parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/0a1efa06-1e15-4a67-8cb4-22a05de1d5df_isrForCrosstalkSources_1123956_27.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/0a1efa06-1e15-4a67-8cb4-22a05de1d5df_isrForCrosstalkSources_1123956_27.stderr parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 139 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 139 has memoization hash 4954305c43be0d881acdcc9aed95fdba parsl.dataflow.memoization INFO: Task 139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a568860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 139 try 0 launched on executor multi with executor id 140 parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/8b68e1fd-6156-4c09-b26d-4f6de53c72ae_isrForCrosstalkSources_1124087_58.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/8b68e1fd-6156-4c09-b26d-4f6de53c72ae_isrForCrosstalkSources_1124087_58.stderr parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 140 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 140 has memoization hash 7ecc9caed903b9fcd07a1f810c8ebc7d parsl.dataflow.memoization INFO: Task 140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a568900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 141 parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/08096885-3c9e-4b84-b8dc-f76de27f94f7_isrForCrosstalkSources_1124087_10.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/08096885-3c9e-4b84-b8dc-f76de27f94f7_isrForCrosstalkSources_1124087_10.stderr parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 141 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 141 has memoization hash 0ca4bb816af0bed93909ab08ca1e9b05 parsl.dataflow.memoization INFO: Task 141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a568a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 141 try 0 launched on executor multi with executor id 142 parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/959ed997-37d3-4cd8-91b1-d644a344d235_isrForCrosstalkSources_1123956_35.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/959ed997-37d3-4cd8-91b1-d644a344d235_isrForCrosstalkSources_1123956_35.stderr parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 142 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 142 has memoization hash 9eca8600115e7ff7a0a1850b90bb2fba parsl.dataflow.memoization INFO: Task 142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a5689a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 143 parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/e2e29434-1d4a-4bd8-9ada-0a59a0497c21_isrForCrosstalkSources_1123958_61.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/e2e29434-1d4a-4bd8-9ada-0a59a0497c21_isrForCrosstalkSources_1123958_61.stderr parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 143 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 143 has memoization hash 23964519869feefe88cf526c52652254 parsl.dataflow.memoization INFO: Task 143 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a568ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 143 try 0 launched on executor multi with executor id 144 parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/6d3a7f42-3a14-4cab-8b2b-abb1ae3119ac_isrForCrosstalkSources_1123958_38.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/6d3a7f42-3a14-4cab-8b2b-abb1ae3119ac_isrForCrosstalkSources_1123958_38.stderr parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 144 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 144 has memoization hash fd74121f62acffae7c967622b4aef5fa parsl.dataflow.memoization INFO: Task 144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a568b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 145 parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/9f46e9e5-5a83-4dcb-853d-bed80078eaa1_isrForCrosstalkSources_1124087_45.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/9f46e9e5-5a83-4dcb-853d-bed80078eaa1_isrForCrosstalkSources_1124087_45.stderr parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 145 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 145 has memoization hash eb2915c62e7f6ddd55d433107bb9c1ab parsl.dataflow.memoization INFO: Task 145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a568c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 145 try 0 launched on executor multi with executor id 146 parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/b70028bf-0624-4190-af8b-0d05cb767470_isrForCrosstalkSources_1123956_1.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/b70028bf-0624-4190-af8b-0d05cb767470_isrForCrosstalkSources_1123956_1.stderr parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 146 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 146 has memoization hash 66a780cdb3792a3c1a4d38354723cc9e parsl.dataflow.memoization INFO: Task 146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a568d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 146 try 0 launched on executor multi with executor id 147 parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/3a84d0f3-35f1-4670-9b0e-3b730323a191_isrForCrosstalkSources_1123956_32.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/3a84d0f3-35f1-4670-9b0e-3b730323a191_isrForCrosstalkSources_1123956_32.stderr parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 147 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 147 has memoization hash 0e243a10f79b9648cf2d41b20f303d38 parsl.dataflow.memoization INFO: Task 147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a568cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 147 try 0 launched on executor multi with executor id 148 parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/820a9ddb-86bc-4668-910d-1cc0fc4151c8_isrForCrosstalkSources_1123958_36.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/820a9ddb-86bc-4668-910d-1cc0fc4151c8_isrForCrosstalkSources_1123958_36.stderr parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 148 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 148 has memoization hash 58350c5dcc6ebe1632f7601d7fd14130 parsl.dataflow.memoization INFO: Task 148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a568e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 149 parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/32484b58-1e6b-4654-b317-cc927b6cb27f_isrForCrosstalkSources_1123956_55.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/32484b58-1e6b-4654-b317-cc927b6cb27f_isrForCrosstalkSources_1123956_55.stderr parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 149 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 149 has memoization hash c8359d1c3d9ea42a0b9ef8c1e93a4e68 parsl.dataflow.memoization INFO: Task 149 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a568ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 149 try 0 launched on executor multi with executor id 150 parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/9bb6f0e8-5d70-4870-8ef7-45af0f1b8b8a_isrForCrosstalkSources_1123956_4.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/9bb6f0e8-5d70-4870-8ef7-45af0f1b8b8a_isrForCrosstalkSources_1123956_4.stderr parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 150 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 150 has memoization hash 030a2c9f0a51b285de92ad4fb444b1cb parsl.dataflow.memoization INFO: Task 150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a568f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 151 parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/a26795b2-e3f9-414d-8a11-ef51f738e4a3_isrForCrosstalkSources_1123956_22.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/a26795b2-e3f9-414d-8a11-ef51f738e4a3_isrForCrosstalkSources_1123956_22.stderr parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 151 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 151 has memoization hash 7f558018fb0c4fc8d605ac89c1ea4894 parsl.dataflow.memoization INFO: Task 151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a569080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 151 try 0 launched on executor multi with executor id 152 parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/43db8f5e-25e8-4402-938f-7a4e38a82915_isrForCrosstalkSources_1123954_18.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/43db8f5e-25e8-4402-938f-7a4e38a82915_isrForCrosstalkSources_1123954_18.stderr parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 152 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 152 has memoization hash c03cb10d37eaae49567126c07df92135 parsl.dataflow.memoization INFO: Task 152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a568fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 152 try 0 launched on executor multi with executor id 153 parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/e7b60cdf-c83f-4d6a-806f-6a04e90c8faf_isrForCrosstalkSources_1123958_39.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/e7b60cdf-c83f-4d6a-806f-6a04e90c8faf_isrForCrosstalkSources_1123958_39.stderr parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 153 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 153 has memoization hash 4b4109bc98344456eec0f62bd16b3059 parsl.dataflow.memoization INFO: Task 153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a569120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 153 try 0 launched on executor multi with executor id 154 parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/485bcfcf-358c-4d2d-aaa6-5dd3d7c56fc1_isrForCrosstalkSources_1123958_43.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/485bcfcf-358c-4d2d-aaa6-5dd3d7c56fc1_isrForCrosstalkSources_1123958_43.stderr parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 154 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 154 has memoization hash df4e39c4900c7149a3f4831bdbc40d73 parsl.dataflow.memoization INFO: Task 154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a5691c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 155 parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/a7844584-a529-4e83-89d0-33e4154104fd_isrForCrosstalkSources_1123954_13.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/a7844584-a529-4e83-89d0-33e4154104fd_isrForCrosstalkSources_1123954_13.stderr parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 155 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 155 has memoization hash aedf537eba954f741fc5fc00264d3088 parsl.dataflow.memoization INFO: Task 155 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a569260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 155 try 0 launched on executor multi with executor id 156 parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/0c6d59c0-52ea-45aa-ae86-c593f652edf1_isrForCrosstalkSources_1123954_52.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/0c6d59c0-52ea-45aa-ae86-c593f652edf1_isrForCrosstalkSources_1123954_52.stderr parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 156 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 156 has memoization hash 5319a30827caab7d6e81a170647ebe6f parsl.dataflow.memoization INFO: Task 156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a5693a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 157 parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/6ddc7722-37df-443f-870a-b6c599c3e933_isrForCrosstalkSources_1123956_18.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/6ddc7722-37df-443f-870a-b6c599c3e933_isrForCrosstalkSources_1123956_18.stderr parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 157 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 157 has memoization hash b65f1158e0ee723af51a6012e3093232 parsl.dataflow.memoization INFO: Task 157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a569300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 03:33:59 proc_lsst.multi:146 [INFO] found job 766 in provider local proc_lsst.multi INFO: found job 766 in provider local parsl.dataflow.dflow INFO: Parsl task 157 try 0 launched on executor multi with executor id 158 parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/6a29e527-1df8-44d2-aa32-bcb59d17b89d_isrForCrosstalkSources_1124087_23.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/6a29e527-1df8-44d2-aa32-bcb59d17b89d_isrForCrosstalkSources_1124087_23.stderr parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 158 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 158 has memoization hash 2413d28d03c329eb7fbe908e6c7897bf parsl.dataflow.memoization INFO: Task 158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a569440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 158 try 0 launched on executor multi with executor id 159 parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/5cb424fa-207a-4a51-b535-f3e26db5d959_isrForCrosstalkSources_1123954_1.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/5cb424fa-207a-4a51-b535-f3e26db5d959_isrForCrosstalkSources_1123954_1.stderr parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 159 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 159 has memoization hash 57d70f4bcee331587be173989ebc2fe5 parsl.dataflow.memoization INFO: Task 159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a5696c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 159 try 0 launched on executor multi with executor id 160 parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/b91ab98a-d51e-4df7-8c98-bf2d2f62216b_isrForCrosstalkSources_1123956_31.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/b91ab98a-d51e-4df7-8c98-bf2d2f62216b_isrForCrosstalkSources_1123956_31.stderr parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 160 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 160 has memoization hash 33e93a1e2b9e5fa9e733524742396b9a parsl.dataflow.memoization INFO: Task 160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a569580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 160 try 0 launched on executor multi with executor id 161 parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/63f85edd-16e7-443d-bfc8-5abce2f1c751_isrForCrosstalkSources_1124087_21.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/63f85edd-16e7-443d-bfc8-5abce2f1c751_isrForCrosstalkSources_1124087_21.stderr parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 161 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 161 has memoization hash 305474b70f56ab7ac8c79215fbb77b6d parsl.dataflow.memoization INFO: Task 161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a569940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 161 try 0 launched on executor multi with executor id 162 parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/e56b7125-1c6f-47ef-ae79-1e88744837a2_isrForCrosstalkSources_1123956_3.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/e56b7125-1c6f-47ef-ae79-1e88744837a2_isrForCrosstalkSources_1123956_3.stderr parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 162 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 162 has memoization hash ad0bcaacc7924825f5642efaf1a807b6 parsl.dataflow.memoization INFO: Task 162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a5698a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 163 parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/7050a1e5-189a-49d7-9515-a87322f13481_isrForCrosstalkSources_1123958_13.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/7050a1e5-189a-49d7-9515-a87322f13481_isrForCrosstalkSources_1123958_13.stderr parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 163 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 163 has memoization hash a7ee8299de06901c4ba941022dd72ba2 parsl.dataflow.memoization INFO: Task 163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a5699e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 163 try 0 launched on executor multi with executor id 164 parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/076e1b5e-0cf3-41a4-aaf7-53fd69b67e33_isrForCrosstalkSources_1124087_19.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/076e1b5e-0cf3-41a4-aaf7-53fd69b67e33_isrForCrosstalkSources_1124087_19.stderr parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 164 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 164 has memoization hash cef4ae02be2240d2a33cc23a477b7237 parsl.dataflow.memoization INFO: Task 164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a569800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 162 parsl.dataflow.strategy DEBUG: Executor multi has 162 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 40 more blocks parsl.executors.status_handling INFO: Scaling out by 40 blocks parsl.executors.status_handling INFO: Allocated block ID 1 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1 2024-03-07 03:34:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 2024-03-07 03:34:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-07 03:34:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 165 parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/4c6fe503-8b8d-479a-8b2c-3d2831114e74_isrForCrosstalkSources_1123954_45.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/4c6fe503-8b8d-479a-8b2c-3d2831114e74_isrForCrosstalkSources_1123954_45.stderr parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 165 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 165 has memoization hash cefa85e85a4cd7af84c40529976c2a27 parsl.dataflow.memoization INFO: Task 165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a569a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 165 try 0 launched on executor multi with executor id 166 parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/3397608d-12da-499e-868d-6bc78715865a_isrForCrosstalkSources_1123954_4.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/3397608d-12da-499e-868d-6bc78715865a_isrForCrosstalkSources_1123954_4.stderr parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 166 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 166 has memoization hash 899880df6e1f7967998a016fda9453a1 parsl.dataflow.memoization INFO: Task 166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a569760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 167 parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/dc8afbc4-8463-4a9f-90a4-6921554b542a_isrForCrosstalkSources_1123956_16.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/dc8afbc4-8463-4a9f-90a4-6921554b542a_isrForCrosstalkSources_1123956_16.stderr parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 167 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 167 has memoization hash 79a4f2874d76529b2219035037ced6f2 parsl.dataflow.memoization INFO: Task 167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a569bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 167 try 0 launched on executor multi with executor id 168 parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/32222add-9750-41e4-8558-d869a6ec042c_isrForCrosstalkSources_1124087_35.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/32222add-9750-41e4-8558-d869a6ec042c_isrForCrosstalkSources_1124087_35.stderr parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 168 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 168 has memoization hash f84defbdd9d09332864ca0f75b9381af parsl.dataflow.memoization INFO: Task 168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a569da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 169 parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/36dd93cc-4e88-49fe-a812-542d9874bb5d_isrForCrosstalkSources_1123954_53.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/36dd93cc-4e88-49fe-a812-542d9874bb5d_isrForCrosstalkSources_1123954_53.stderr parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 169 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 169 has memoization hash 2d371c9bf167788268ce4fbefc9bbfcd parsl.dataflow.memoization INFO: Task 169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 169 try 0 launched on executor multi with executor id 170 parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/61cd4253-fe6c-4bff-92a8-1a628353454a_isrForCrosstalkSources_1123958_33.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/61cd4253-fe6c-4bff-92a8-1a628353454a_isrForCrosstalkSources_1123958_33.stderr parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 170 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 170 has memoization hash 707e8096650c82a624d8a3c1ca07f38c parsl.dataflow.memoization INFO: Task 170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a569620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 170 try 0 launched on executor multi with executor id 171 parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/e7edfc4c-be51-4249-930e-5c696dfd32d9_isrForCrosstalkSources_1123958_14.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/e7edfc4c-be51-4249-930e-5c696dfd32d9_isrForCrosstalkSources_1123958_14.stderr parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 171 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 171 has memoization hash 1ac3c4dc4cbe71b9564f9b25f537d484 parsl.dataflow.memoization INFO: Task 171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 171 try 0 launched on executor multi with executor id 172 parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/e9f4dc88-b052-456a-93db-28da973ae5cf_isrForCrosstalkSources_1123958_44.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/e9f4dc88-b052-456a-93db-28da973ae5cf_isrForCrosstalkSources_1123958_44.stderr parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 172 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 172 has memoization hash c6c4ae65efdfa6253bc0e3ecab19b8b5 parsl.dataflow.memoization INFO: Task 172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a569b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 173 parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/72e88f04-129e-42f2-9a38-522daa6d47b4_isrForCrosstalkSources_1123958_24.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/72e88f04-129e-42f2-9a38-522daa6d47b4_isrForCrosstalkSources_1123958_24.stderr parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 173 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 173 has memoization hash a50e59004db91a56eb0828d17c603562 parsl.dataflow.memoization INFO: Task 173 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e8793c40e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 173 try 0 launched on executor multi with executor id 174 parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/79f6a555-7b9c-45e9-8e3c-0fe865f5b3fc_isrForCrosstalkSources_1124087_38.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/79f6a555-7b9c-45e9-8e3c-0fe865f5b3fc_isrForCrosstalkSources_1124087_38.stderr parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 174 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 174 has memoization hash 1dead3ecedc4d3b85fff208e4e07cc3f parsl.dataflow.memoization INFO: Task 174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a569f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 175 parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/5875cd7a-a43c-4070-b518-72819133dfce_isrForCrosstalkSources_1123956_61.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/5875cd7a-a43c-4070-b518-72819133dfce_isrForCrosstalkSources_1123956_61.stderr parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 175 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 175 has memoization hash 21295bfd0597173015695feac0df41b7 parsl.dataflow.memoization INFO: Task 175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 175 try 0 launched on executor multi with executor id 176 parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/ee36f9bd-5e7e-48a4-95f1-b38aeec8f20a_isrForCrosstalkSources_1123956_10.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/ee36f9bd-5e7e-48a4-95f1-b38aeec8f20a_isrForCrosstalkSources_1123956_10.stderr parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 176 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 176 has memoization hash 145344565f7dca68864dad336823257c parsl.dataflow.memoization INFO: Task 176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 177 parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/06a65fda-76f8-4170-83a2-bc68462145e5_isrForCrosstalkSources_1123956_59.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/06a65fda-76f8-4170-83a2-bc68462145e5_isrForCrosstalkSources_1123956_59.stderr parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 177 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 177 has memoization hash 6180d0f0fffe977ad2e648f31e73550d parsl.dataflow.memoization INFO: Task 177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 177 try 0 launched on executor multi with executor id 178 parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/bf2aebb3-70dc-4ee5-94d6-a3cf8372fc0d_isrForCrosstalkSources_1123954_7.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/bf2aebb3-70dc-4ee5-94d6-a3cf8372fc0d_isrForCrosstalkSources_1123954_7.stderr parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 178 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 178 has memoization hash e624951a502924ae28b41d9ba00e8358 parsl.dataflow.memoization INFO: Task 178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a569ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 179 parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/69518b60-790c-40a5-ba10-165a89a77625_isrForCrosstalkSources_1124087_9.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/69518b60-790c-40a5-ba10-165a89a77625_isrForCrosstalkSources_1124087_9.stderr parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 179 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 179 has memoization hash 916f494e6102a0b953c64e9de58b64f1 parsl.dataflow.memoization INFO: Task 179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 179 try 0 launched on executor multi with executor id 180 parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/b6dd5e9f-c2d0-49b9-8078-00b3ff7ddb74_isrForCrosstalkSources_1123958_62.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/b6dd5e9f-c2d0-49b9-8078-00b3ff7ddb74_isrForCrosstalkSources_1123958_62.stderr parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 180 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 180 has memoization hash 1059fdfd79983ed3347a503881ac0833 parsl.dataflow.memoization INFO: Task 180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 181 parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/9c1eba33-6446-4ff9-8e8e-a390300a2d7b_isrForCrosstalkSources_1124087_26.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/9c1eba33-6446-4ff9-8e8e-a390300a2d7b_isrForCrosstalkSources_1124087_26.stderr parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 181 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 181 has memoization hash ddc3a3de14e22bbffcd5ff4cf24c8a2e parsl.dataflow.memoization INFO: Task 181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 181 try 0 launched on executor multi with executor id 182 parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/fa9c00b5-c2dc-4c99-833e-2c489f5f8668_isrForCrosstalkSources_1124087_53.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/fa9c00b5-c2dc-4c99-833e-2c489f5f8668_isrForCrosstalkSources_1124087_53.stderr parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 182 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 182 has memoization hash bc3a8d647973242d0919e2a2fa10fa4e parsl.dataflow.memoization INFO: Task 182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 183 parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/2ee915de-2511-4d22-af35-6ff774e6b256_isrForCrosstalkSources_1123956_57.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/2ee915de-2511-4d22-af35-6ff774e6b256_isrForCrosstalkSources_1123956_57.stderr parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 183 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 183 has memoization hash 072c20f5a1c89a5db6fbef6e88d174de parsl.dataflow.memoization INFO: Task 183 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 183 try 0 launched on executor multi with executor id 184 parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/a50f25c0-e587-4077-a5f8-3076b5133993_isrForCrosstalkSources_1124087_36.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/a50f25c0-e587-4077-a5f8-3076b5133993_isrForCrosstalkSources_1124087_36.stderr parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 184 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 184 has memoization hash f9fb5def48528249b95dd9ed15724e39 parsl.dataflow.memoization INFO: Task 184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56a660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 185 parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/19ce9793-ad83-4eb2-8451-6812d0bf21e0_isrForCrosstalkSources_1123954_8.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/19ce9793-ad83-4eb2-8451-6812d0bf21e0_isrForCrosstalkSources_1123954_8.stderr parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 185 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 185 has memoization hash 51227af4c82fa2fce813755ef7fc0b47 parsl.dataflow.memoization INFO: Task 185 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 185 try 0 launched on executor multi with executor id 186 parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/6bbea2a1-da77-4756-b6cc-6a7c19d434ab_isrForCrosstalkSources_1123956_49.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/6bbea2a1-da77-4756-b6cc-6a7c19d434ab_isrForCrosstalkSources_1123956_49.stderr parsl.dataflow.dflow DEBUG: Task 186 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 186 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 186 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 186 has memoization hash 327e9513e29ef0dc57f944d435c5d44c parsl.dataflow.memoization INFO: Task 186 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 186 try 0 launched on executor multi with executor id 187 parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/4cf28a67-fc6c-4a59-838f-3234e462a29d_isrForCrosstalkSources_1123956_51.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/4cf28a67-fc6c-4a59-838f-3234e462a29d_isrForCrosstalkSources_1123956_51.stderr parsl.dataflow.dflow DEBUG: Task 187 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 187 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 187 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 187 has memoization hash c215221b0862892b49390ecbbc805fa6 parsl.dataflow.memoization INFO: Task 187 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 187 try 0 launched on executor multi with executor id 188 parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/b93ca3b6-8fbe-4b6b-ac17-182185d759f1_isrForCrosstalkSources_1123954_17.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/b93ca3b6-8fbe-4b6b-ac17-182185d759f1_isrForCrosstalkSources_1123954_17.stderr parsl.dataflow.dflow DEBUG: Task 188 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 188 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 188 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 188 has memoization hash b9b84c4b2a7c5a2c2a11a70709fef6b6 parsl.dataflow.memoization INFO: Task 188 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 188 try 0 launched on executor multi with executor id 189 parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/3e454e00-9965-48ad-97e4-718f97943002_isrForCrosstalkSources_1124087_61.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/3e454e00-9965-48ad-97e4-718f97943002_isrForCrosstalkSources_1124087_61.stderr parsl.dataflow.dflow DEBUG: Task 189 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 189 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 189 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 189 has memoization hash 508a9cee90a9b0ce22e5c718c1a42643 parsl.dataflow.memoization INFO: Task 189 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 189 try 0 launched on executor multi with executor id 190 parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/afa3fcd5-c210-4377-a16f-2aecb6406197_isrForCrosstalkSources_1123958_19.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/afa3fcd5-c210-4377-a16f-2aecb6406197_isrForCrosstalkSources_1123958_19.stderr parsl.dataflow.dflow DEBUG: Task 190 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 190 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 190 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 190 has memoization hash 0d24f3053cafdb82c043e1facfff0a8a parsl.dataflow.memoization INFO: Task 190 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 190 try 0 launched on executor multi with executor id 191 parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/51f10e95-5757-4ea4-9a53-1deed5b2164a_isrForCrosstalkSources_1123958_29.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/51f10e95-5757-4ea4-9a53-1deed5b2164a_isrForCrosstalkSources_1123958_29.stderr parsl.dataflow.dflow DEBUG: Task 191 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 191 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 191 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 191 has memoization hash 154b9d7c831ec00b898b40604fdfa6df parsl.dataflow.memoization INFO: Task 191 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 191 try 0 launched on executor multi with executor id 192 parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/9ce31ac4-4d09-4644-b2ef-66e15db0af37_isrForCrosstalkSources_1123958_40.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/9ce31ac4-4d09-4644-b2ef-66e15db0af37_isrForCrosstalkSources_1123958_40.stderr parsl.dataflow.dflow DEBUG: Task 192 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 192 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 192 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 192 has memoization hash 6d4b165a15839bc58fdbbab2a9873a11 parsl.dataflow.memoization INFO: Task 192 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 192 try 0 launched on executor multi with executor id 193 parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/867ddbde-dad0-4d91-8719-8c567657a4d7_isrForCrosstalkSources_1124087_34.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/867ddbde-dad0-4d91-8719-8c567657a4d7_isrForCrosstalkSources_1124087_34.stderr parsl.dataflow.dflow DEBUG: Task 193 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 193 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 193 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 193 has memoization hash 67dacfb52251eb0b96dd98e85a1a250a parsl.dataflow.memoization INFO: Task 193 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 193 try 0 launched on executor multi with executor id 194 parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/3b18fd6a-f0d1-4b6e-b05a-f6c877c7a402_isrForCrosstalkSources_1123958_21.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/3b18fd6a-f0d1-4b6e-b05a-f6c877c7a402_isrForCrosstalkSources_1123958_21.stderr parsl.dataflow.dflow DEBUG: Task 194 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 194 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 194 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 194 has memoization hash 5c84b911ed8a24950ecddce9382aa557 parsl.dataflow.memoization INFO: Task 194 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 194 try 0 launched on executor multi with executor id 195 parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/86543f81-6ec7-40c5-8cc1-897dc4eb2339_isrForCrosstalkSources_1123958_3.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/86543f81-6ec7-40c5-8cc1-897dc4eb2339_isrForCrosstalkSources_1123958_3.stderr parsl.dataflow.dflow DEBUG: Task 195 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 195 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 195 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 195 has memoization hash ceb50c6c21265b054f3c40d208fa624a parsl.dataflow.memoization INFO: Task 195 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 195 try 0 launched on executor multi with executor id 196 parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/f3e48781-ad00-44eb-ae06-a34171c071ee_isrForCrosstalkSources_1124087_60.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/f3e48781-ad00-44eb-ae06-a34171c071ee_isrForCrosstalkSources_1124087_60.stderr parsl.dataflow.dflow DEBUG: Task 196 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 196 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 196 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 196 has memoization hash f9e5023137e11ee0c79707b9533d6869 parsl.dataflow.memoization INFO: Task 196 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 03:34:00 proc_lsst.multi:170 [INFO] [multi] job_id 16884172 proc_lsst.multi INFO: [multi] job_id 16884172 2024-03-07 03:34:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:00 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884172 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884172 parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 16884172 parsl.executors.status_handling INFO: Allocated block ID 2 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2 2024-03-07 03:34:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 2024-03-07 03:34:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 196 try 0 launched on executor multi with executor id 197 parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/16592b77-0e74-458c-b0ae-816c226f9d1e_isrForCrosstalkSources_1123958_52.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/16592b77-0e74-458c-b0ae-816c226f9d1e_isrForCrosstalkSources_1123958_52.stderr parsl.dataflow.dflow DEBUG: Task 197 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 197 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 197 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 197 has memoization hash d75c68f195ef9b4de5802c32f3528606 parsl.dataflow.memoization INFO: Task 197 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 197 try 0 launched on executor multi with executor id 198 parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/5084079e-67e5-4a28-8e6c-331dd2451632_isrForCrosstalkSources_1123954_59.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/5084079e-67e5-4a28-8e6c-331dd2451632_isrForCrosstalkSources_1123954_59.stderr parsl.dataflow.dflow DEBUG: Task 198 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 198 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 198 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 198 has memoization hash 800f92d7171efac0c5433de0ead923ba parsl.dataflow.memoization INFO: Task 198 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 198 try 0 launched on executor multi with executor id 199 parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/d297b500-bf99-483c-bd86-32e3981ec974_isrForCrosstalkSources_1123954_25.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/d297b500-bf99-483c-bd86-32e3981ec974_isrForCrosstalkSources_1123954_25.stderr parsl.dataflow.dflow DEBUG: Task 199 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 199 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 199 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 199 has memoization hash 172bd129de1833e8e8f82dc3881a6a99 parsl.dataflow.memoization INFO: Task 199 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a569c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 199 try 0 launched on executor multi with executor id 200 parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/3e386935-b586-497b-ad94-1cae405043a2_isrForCrosstalkSources_1123954_56.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/3e386935-b586-497b-ad94-1cae405043a2_isrForCrosstalkSources_1123954_56.stderr parsl.dataflow.dflow DEBUG: Task 200 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 200 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 200 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 200 has memoization hash ad3a53a6373e83d7e8157f05d9a663ab parsl.dataflow.memoization INFO: Task 200 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56afc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 200 try 0 launched on executor multi with executor id 201 parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/3d39beea-21e2-4d7e-876b-8bc04c95998d_isrForCrosstalkSources_1124087_52.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/3d39beea-21e2-4d7e-876b-8bc04c95998d_isrForCrosstalkSources_1124087_52.stderr parsl.dataflow.dflow DEBUG: Task 201 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 201 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 201 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 201 has memoization hash 478af14e4fa9af6397f741c3bb7a3fc4 parsl.dataflow.memoization INFO: Task 201 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a569e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 201 try 0 launched on executor multi with executor id 202 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/4f13638b-8080-4e43-a286-cd9b0065b760_isrForCrosstalkSources_1123958_31.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/4f13638b-8080-4e43-a286-cd9b0065b760_isrForCrosstalkSources_1123958_31.stderr parsl.dataflow.dflow DEBUG: Task 202 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 202 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 202 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 202 has memoization hash fb6163e3968e5182fbd3eb4e3e8f753f parsl.dataflow.memoization INFO: Task 202 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e8793c6520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 202 try 0 launched on executor multi with executor id 203 parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/f18f6540-5cdf-4599-a087-ca00a82e368c_isrForCrosstalkSources_1123958_41.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/f18f6540-5cdf-4599-a087-ca00a82e368c_isrForCrosstalkSources_1123958_41.stderr parsl.dataflow.dflow DEBUG: Task 203 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 203 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 203 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 203 has memoization hash 0d970a1d873a940d204659bc32ed07fe parsl.dataflow.memoization INFO: Task 203 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e8793c6480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 203 try 0 launched on executor multi with executor id 204 parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/799e5421-61c6-4b86-9ad1-a22582b8a548_isrForCrosstalkSources_1123958_60.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/799e5421-61c6-4b86-9ad1-a22582b8a548_isrForCrosstalkSources_1123958_60.stderr parsl.dataflow.dflow DEBUG: Task 204 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 204 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 204 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 204 has memoization hash 59bd213208706378ece6757b18374b49 parsl.dataflow.memoization INFO: Task 204 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e8793c65c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 204 try 0 launched on executor multi with executor id 205 parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/813e9cd0-a9c5-4379-9afd-fd3ad32257b0_isrForCrosstalkSources_1123954_29.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/813e9cd0-a9c5-4379-9afd-fd3ad32257b0_isrForCrosstalkSources_1123954_29.stderr parsl.dataflow.dflow DEBUG: Task 205 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 205 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 205 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 205 has memoization hash 680134c621504df9c24290d5092f7f55 parsl.dataflow.memoization INFO: Task 205 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 205 try 0 launched on executor multi with executor id 206 parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/8c09b022-90e6-4b21-b6a9-226dd17e5471_isrForCrosstalkSources_1123956_14.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/8c09b022-90e6-4b21-b6a9-226dd17e5471_isrForCrosstalkSources_1123956_14.stderr parsl.dataflow.dflow DEBUG: Task 206 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 206 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 206 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 206 has memoization hash a676417a7ee2511286bce71f920cac7b parsl.dataflow.memoization INFO: Task 206 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 206 try 0 launched on executor multi with executor id 207 parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/48355fd9-0bc7-40af-acc5-67d82003d614_isrForCrosstalkSources_1123958_42.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/48355fd9-0bc7-40af-acc5-67d82003d614_isrForCrosstalkSources_1123958_42.stderr parsl.dataflow.dflow DEBUG: Task 207 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 207 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 207 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 207 has memoization hash 294293750313be28a8523109830170b2 parsl.dataflow.memoization INFO: Task 207 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56b2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 207 try 0 launched on executor multi with executor id 208 parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/b4b1e2fc-82f4-4e3d-bd21-d94ac07b818c_isrForCrosstalkSources_1123954_28.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/b4b1e2fc-82f4-4e3d-bd21-d94ac07b818c_isrForCrosstalkSources_1123954_28.stderr parsl.dataflow.dflow DEBUG: Task 208 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 208 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 208 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 208 has memoization hash 679ce072931344c508a595989f996436 parsl.dataflow.memoization INFO: Task 208 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 208 try 0 launched on executor multi with executor id 209 parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/8bf2b414-3e56-41be-bf93-8edb5eddb992_isrForCrosstalkSources_1123958_37.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/8bf2b414-3e56-41be-bf93-8edb5eddb992_isrForCrosstalkSources_1123958_37.stderr parsl.dataflow.dflow DEBUG: Task 209 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 209 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 209 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 209 has memoization hash 80169c653a570c22b383752c99113c2e parsl.dataflow.memoization INFO: Task 209 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 209 try 0 launched on executor multi with executor id 210 parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/c281b501-0bce-4c32-9706-dccedb1111eb_isrForCrosstalkSources_1124087_13.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/c281b501-0bce-4c32-9706-dccedb1111eb_isrForCrosstalkSources_1124087_13.stderr parsl.dataflow.dflow DEBUG: Task 210 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 210 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 210 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 210 has memoization hash 2d4ad50cfdbc3745b7ae83a27e609998 parsl.dataflow.memoization INFO: Task 210 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 210 try 0 launched on executor multi with executor id 211 parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/5e2e2e05-a61e-4c94-aa28-b208cf7deef7_isrForCrosstalkSources_1123954_49.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/5e2e2e05-a61e-4c94-aa28-b208cf7deef7_isrForCrosstalkSources_1123954_49.stderr parsl.dataflow.dflow DEBUG: Task 211 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 211 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 211 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 211 has memoization hash eac4b40767a07feabebc58754b1dc75b parsl.dataflow.memoization INFO: Task 211 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 211 try 0 launched on executor multi with executor id 212 parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/96dcbbfa-b490-4e0b-ae0e-e5dcee6d4b34_isrForCrosstalkSources_1124087_3.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/96dcbbfa-b490-4e0b-ae0e-e5dcee6d4b34_isrForCrosstalkSources_1124087_3.stderr parsl.dataflow.dflow DEBUG: Task 212 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 212 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 212 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 212 has memoization hash dd73e99871d6ea1b5063cbc655b6d872 parsl.dataflow.memoization INFO: Task 212 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 212 try 0 launched on executor multi with executor id 213 parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/03194941-efbf-4417-8dff-df1208ffecfd_isrForCrosstalkSources_1123956_9.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/03194941-efbf-4417-8dff-df1208ffecfd_isrForCrosstalkSources_1123956_9.stderr parsl.dataflow.dflow DEBUG: Task 213 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 213 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 213 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 213 has memoization hash 4c24b3ba9b21147bd10d534e4a6c4bc4 parsl.dataflow.memoization INFO: Task 213 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 213 try 0 launched on executor multi with executor id 214 parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/0548b533-9b78-4477-bc95-81408dfd3dfd_isrForCrosstalkSources_1124087_55.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/0548b533-9b78-4477-bc95-81408dfd3dfd_isrForCrosstalkSources_1124087_55.stderr parsl.dataflow.dflow DEBUG: Task 214 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 214 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 214 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 214 has memoization hash 785c8b0588283672dc1e22b85278f007 parsl.dataflow.memoization INFO: Task 214 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56b600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 214 try 0 launched on executor multi with executor id 215 parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/3827a433-3b5a-4199-acd8-4e3b77d1c0e4_isrForCrosstalkSources_1123956_46.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/3827a433-3b5a-4199-acd8-4e3b77d1c0e4_isrForCrosstalkSources_1123956_46.stderr parsl.dataflow.dflow DEBUG: Task 215 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 215 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 215 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 215 has memoization hash 12334ec4c541b35b010476625a72e188 parsl.dataflow.memoization INFO: Task 215 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 215 try 0 launched on executor multi with executor id 216 parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/dcb3fb92-2730-4537-ba80-49611876fdca_isrForCrosstalkSources_1123954_44.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/dcb3fb92-2730-4537-ba80-49611876fdca_isrForCrosstalkSources_1123954_44.stderr parsl.dataflow.dflow DEBUG: Task 216 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 216 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 216 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 216 has memoization hash 5ca69430d9708157dfe9675cb4eda1b2 parsl.dataflow.memoization INFO: Task 216 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 216 try 0 launched on executor multi with executor id 217 parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/0978e9e6-ea35-4e08-9490-983681677f7a_isrForCrosstalkSources_1123958_15.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/0978e9e6-ea35-4e08-9490-983681677f7a_isrForCrosstalkSources_1123958_15.stderr parsl.dataflow.dflow DEBUG: Task 217 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 217 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 217 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 217 has memoization hash 991732ca8fb5e8205673905aff276d3c parsl.dataflow.memoization INFO: Task 217 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 217 try 0 launched on executor multi with executor id 218 parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/1334ee97-84bb-4ffc-be56-c5f45b9d9bd8_isrForCrosstalkSources_1123956_48.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/1334ee97-84bb-4ffc-be56-c5f45b9d9bd8_isrForCrosstalkSources_1123956_48.stderr parsl.dataflow.dflow DEBUG: Task 218 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 218 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 218 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 218 has memoization hash 7847dee8f9a178ca74b875d34421e447 parsl.dataflow.memoization INFO: Task 218 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 218 try 0 launched on executor multi with executor id 219 parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/a2699cca-e736-46ab-a92c-798a3f1cc841_isrForCrosstalkSources_1124087_41.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/a2699cca-e736-46ab-a92c-798a3f1cc841_isrForCrosstalkSources_1124087_41.stderr parsl.dataflow.dflow DEBUG: Task 219 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 219 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 219 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 219 has memoization hash 64e9ac0fc5ab0c550fe3edbe5a30c2f6 parsl.dataflow.memoization INFO: Task 219 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 219 try 0 launched on executor multi with executor id 220 parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/0b7c0643-7922-4ea4-b46f-ffca298258a0_isrForCrosstalkSources_1123958_51.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/0b7c0643-7922-4ea4-b46f-ffca298258a0_isrForCrosstalkSources_1123958_51.stderr parsl.dataflow.dflow DEBUG: Task 220 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 220 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 220 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 220 has memoization hash bfc011f73fdbf3fc67cb796c4c376668 parsl.dataflow.memoization INFO: Task 220 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56ba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 220 try 0 launched on executor multi with executor id 221 parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/b765faf0-acd4-429a-85e2-34db494106c8_isrForCrosstalkSources_1123956_52.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/b765faf0-acd4-429a-85e2-34db494106c8_isrForCrosstalkSources_1123956_52.stderr parsl.dataflow.dflow DEBUG: Task 221 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 221 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 221 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 221 has memoization hash 11330828fe9444165d1deff355b3bf1a parsl.dataflow.memoization INFO: Task 221 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56b9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 221 try 0 launched on executor multi with executor id 222 parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/b3450c1d-0001-4f93-a2ac-5498dcc86302_isrForCrosstalkSources_1123954_35.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/b3450c1d-0001-4f93-a2ac-5498dcc86302_isrForCrosstalkSources_1123954_35.stderr parsl.dataflow.dflow DEBUG: Task 222 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 222 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 222 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 222 has memoization hash f42706d190b3414c362a6d9138901f17 parsl.dataflow.memoization INFO: Task 222 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 222 try 0 launched on executor multi with executor id 223 parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/fbbf51cb-65b5-4c12-bf45-897c556694f7_isrForCrosstalkSources_1123954_58.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/fbbf51cb-65b5-4c12-bf45-897c556694f7_isrForCrosstalkSources_1123954_58.stderr parsl.dataflow.dflow DEBUG: Task 223 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 223 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 223 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 223 has memoization hash 58e48215da2cf8e9570058f6bcc1b2e6 parsl.dataflow.memoization INFO: Task 223 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 223 try 0 launched on executor multi with executor id 224 parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/eb005449-2d24-41b2-9e1a-4dc0f0bf15fe_isrForCrosstalkSources_1123956_12.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/eb005449-2d24-41b2-9e1a-4dc0f0bf15fe_isrForCrosstalkSources_1123956_12.stderr parsl.dataflow.dflow DEBUG: Task 224 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 224 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 224 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 224 has memoization hash a039309851f3397b60bc852e74067dbf parsl.dataflow.memoization INFO: Task 224 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 224 try 0 launched on executor multi with executor id 225 parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/e70a4d40-d3f0-4b9c-9b58-ee7e9b5b34b0_isrForCrosstalkSources_1123958_6.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/e70a4d40-d3f0-4b9c-9b58-ee7e9b5b34b0_isrForCrosstalkSources_1123958_6.stderr parsl.dataflow.dflow DEBUG: Task 225 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 225 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 225 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 225 has memoization hash 970bfdaf30935549f3bfa36209416b9f parsl.dataflow.memoization INFO: Task 225 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 225 try 0 launched on executor multi with executor id 226 parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/e5a36366-cab6-4e95-ab00-8ff3c50e98e9_isrForCrosstalkSources_1123958_35.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/e5a36366-cab6-4e95-ab00-8ff3c50e98e9_isrForCrosstalkSources_1123958_35.stderr parsl.dataflow.dflow DEBUG: Task 226 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 226 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 226 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 226 has memoization hash c25967c623ed784a46bfb834577c2a18 parsl.dataflow.memoization INFO: Task 226 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 226 try 0 launched on executor multi with executor id 227 parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/e83904b9-f899-417b-b6c7-511e6dd7d9b8_isrForCrosstalkSources_1123956_60.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/e83904b9-f899-417b-b6c7-511e6dd7d9b8_isrForCrosstalkSources_1123956_60.stderr parsl.dataflow.dflow DEBUG: Task 227 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 227 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 227 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 227 has memoization hash 3f332a905b59259226f24766ab4c9d29 parsl.dataflow.memoization INFO: Task 227 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e8793c6340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 227 try 0 launched on executor multi with executor id 228 parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/c1ec565a-773b-4689-a741-b8bc5bfc0b50_isrForCrosstalkSources_1123958_32.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/c1ec565a-773b-4689-a741-b8bc5bfc0b50_isrForCrosstalkSources_1123958_32.stderr parsl.dataflow.dflow DEBUG: Task 228 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 228 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 228 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 228 has memoization hash fe5200417c9432cc7b0da372c4de0f28 parsl.dataflow.memoization INFO: Task 228 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e8793c6660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 228 try 0 launched on executor multi with executor id 229 parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/294c06fa-fd04-44be-881d-c5ab168e7fd4_isrForCrosstalkSources_1124087_39.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/294c06fa-fd04-44be-881d-c5ab168e7fd4_isrForCrosstalkSources_1124087_39.stderr parsl.dataflow.dflow DEBUG: Task 229 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 229 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 229 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 229 has memoization hash c9fc24a1f784eb946ca49be1816f78ff parsl.dataflow.memoization INFO: Task 229 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e8793c62a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 229 try 0 launched on executor multi with executor id 230 parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/2a9e3705-a9b3-458c-b32f-7eb0c9c6c962_isrForCrosstalkSources_1124087_49.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/2a9e3705-a9b3-458c-b32f-7eb0c9c6c962_isrForCrosstalkSources_1124087_49.stderr parsl.dataflow.dflow DEBUG: Task 230 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 230 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 230 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 230 has memoization hash 3c786d170a9686b666956482315ba0e6 parsl.dataflow.memoization INFO: Task 230 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e8793c4180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 230 try 0 launched on executor multi with executor id 231 parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/bf213554-e7f6-4647-8b58-45a4d9a9a368_isrForCrosstalkSources_1123958_47.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/bf213554-e7f6-4647-8b58-45a4d9a9a368_isrForCrosstalkSources_1123958_47.stderr parsl.dataflow.dflow DEBUG: Task 231 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 231 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 231 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 231 has memoization hash 30ac6968a39f214c30413c7ae66864e3 parsl.dataflow.memoization INFO: Task 231 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e8793c42c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 231 try 0 launched on executor multi with executor id 232 parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/16654c15-77c3-46d2-af84-a48e6b76433a_isrForCrosstalkSources_1123956_8.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/16654c15-77c3-46d2-af84-a48e6b76433a_isrForCrosstalkSources_1123956_8.stderr 2024-03-07 03:34:00 proc_lsst.multi:170 [INFO] [multi] job_id 16884173 parsl.dataflow.dflow DEBUG: Task 232 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 232 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 232 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 232 has memoization hash 53fbe425acf22a8f9fea030a3f57cbe6 parsl.dataflow.memoization INFO: Task 232 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 232 try 0 launched on executor multi with executor id 233 parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/64120a94-6f9f-443c-a831-6bb9c29165da_isrForCrosstalkSources_1124087_50.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/64120a94-6f9f-443c-a831-6bb9c29165da_isrForCrosstalkSources_1124087_50.stderr parsl.dataflow.dflow DEBUG: Task 233 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 233 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 233 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 233 has memoization hash fd187def9aa7b12f3c1fc3a949c9d24e parsl.dataflow.memoization INFO: Task 233 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 233 try 0 launched on executor multi with executor id 234 parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/ea5d1c56-bbad-4220-b789-82785b2e269d_isrForCrosstalkSources_1124087_24.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/ea5d1c56-bbad-4220-b789-82785b2e269d_isrForCrosstalkSources_1124087_24.stderr proc_lsst.multi INFO: [multi] job_id 16884173 2024-03-07 03:34:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 03:34:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:00 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884173 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884173 parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 16884173 parsl.executors.status_handling INFO: Allocated block ID 3 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3 2024-03-07 03:34:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 2024-03-07 03:34:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 03:34:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 234 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 234 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 234 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 234 has memoization hash 97eee4528c737b04b402ae425a37cb8e parsl.dataflow.memoization INFO: Task 234 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e87a56a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 234 try 0 launched on executor multi with executor id 235 parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/3f2df7ee-b226-4fe8-b93e-251affc6888a_isrForCrosstalkSources_1123958_58.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/3f2df7ee-b226-4fe8-b93e-251affc6888a_isrForCrosstalkSources_1123958_58.stderr parsl.dataflow.dflow DEBUG: Task 235 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 235 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 235 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 235 has memoization hash 64a5ca4c7ff3bd2a49c09e0cf060f76a parsl.dataflow.memoization INFO: Task 235 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e8793c63e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 235 try 0 launched on executor multi with executor id 236 parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/85ca26dc-37ab-4b03-ab76-a6da4ab961a5_isrForCrosstalkSources_1123954_40.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/85ca26dc-37ab-4b03-ab76-a6da4ab961a5_isrForCrosstalkSources_1123954_40.stderr parsl.dataflow.dflow DEBUG: Task 236 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 236 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 236 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 236 has memoization hash f55bc3614cea1f70570a75a73128200d parsl.dataflow.memoization INFO: Task 236 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e8793c4040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 236 try 0 launched on executor multi with executor id 237 parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/cfb20153-8d2a-421d-892d-6d9cdfed28c4_isrForCrosstalkSources_1123954_26.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/cfb20153-8d2a-421d-892d-6d9cdfed28c4_isrForCrosstalkSources_1123954_26.stderr parsl.dataflow.dflow DEBUG: Task 237 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 237 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 237 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 237 has memoization hash 94f78930f3e624689de7274f52c8ecf0 parsl.dataflow.memoization INFO: Task 237 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e8793c4360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 237 try 0 launched on executor multi with executor id 238 parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/eeea1d88-dc49-4dbe-9da5-a42dcab5e9ba_isrForCrosstalkSources_1124087_31.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/eeea1d88-dc49-4dbe-9da5-a42dcab5e9ba_isrForCrosstalkSources_1124087_31.stderr parsl.dataflow.dflow DEBUG: Task 238 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 238 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 238 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 238 has memoization hash 50aa0c2e2b64a39e27116c75d138dd00 parsl.dataflow.memoization INFO: Task 238 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e8793c44a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 238 try 0 launched on executor multi with executor id 239 parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/1f99a5c0-b51a-477e-862a-f87901b0ebef_isrForCrosstalkSources_1123956_36.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/1f99a5c0-b51a-477e-862a-f87901b0ebef_isrForCrosstalkSources_1123956_36.stderr parsl.dataflow.dflow DEBUG: Task 239 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 239 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 239 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.memoization DEBUG: Task 239 has memoization hash e7af1bef951d9ba77d62835e0fbbd70e parsl.dataflow.memoization INFO: Task 239 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e8793c4540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 239 try 0 launched on executor multi with executor id 240 parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/6dde717a-411b-4f04-91b0-05cce6a4b3ac_isrForCrosstalkSources_1123954_57.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/6dde717a-411b-4f04-91b0-05cce6a4b3ac_isrForCrosstalkSources_1123954_57.stderr parsl.dataflow.dflow DEBUG: Task 240 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 240 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 240 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 240 has memoization hash ee47b7731e1fec21c197c0f4bd96fb53 parsl.dataflow.memoization INFO: Task 240 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e8793c4220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 240 try 0 launched on executor multi with executor id 241 parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/9b0e094e-c248-4117-ac9d-6e23ad30ef0a_isrForCrosstalkSources_1123958_27.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/9b0e094e-c248-4117-ac9d-6e23ad30ef0a_isrForCrosstalkSources_1123958_27.stderr parsl.dataflow.dflow DEBUG: Task 241 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 241 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 241 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 241 has memoization hash f1f9eb9ea7da6b31328515b1966c28d9 parsl.dataflow.memoization INFO: Task 241 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e8793c45e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 241 try 0 launched on executor multi with executor id 242 parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/bd236e00-9bca-4296-a3f4-b721ccabeb0b_isrForCrosstalkSources_1123956_28.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/bd236e00-9bca-4296-a3f4-b721ccabeb0b_isrForCrosstalkSources_1123956_28.stderr parsl.dataflow.dflow DEBUG: Task 242 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 242 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 242 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 242 has memoization hash 70f7cfbb953f931c503be8dfbb4bffd6 parsl.dataflow.memoization INFO: Task 242 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e8793c4680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 242 try 0 launched on executor multi with executor id 243 parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/abb236fd-8273-4d64-a7ce-65ed3b84b8a2_isrForCrosstalkSources_1124087_37.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/abb236fd-8273-4d64-a7ce-65ed3b84b8a2_isrForCrosstalkSources_1124087_37.stderr parsl.dataflow.dflow DEBUG: Task 243 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 243 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 243 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 243 has memoization hash d6086b6d5ee38d3f0f0de63ad651e02d parsl.dataflow.memoization INFO: Task 243 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e8793c47c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 243 try 0 launched on executor multi with executor id 244 parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/9f0639a0-0e87-4019-a70c-8b22f474df01_isrForCrosstalkSources_1124087_2.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/9f0639a0-0e87-4019-a70c-8b22f474df01_isrForCrosstalkSources_1124087_2.stderr parsl.dataflow.dflow DEBUG: Task 244 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 244 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 244 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 244 has memoization hash 674e343cd646b9e6bbf66984982a23c0 parsl.dataflow.memoization INFO: Task 244 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e8793c4720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 244 try 0 launched on executor multi with executor id 245 parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/0251cf6d-dd64-479b-8b9a-edbd7169d519_isrForCrosstalkSources_1123956_41.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/0251cf6d-dd64-479b-8b9a-edbd7169d519_isrForCrosstalkSources_1123956_41.stderr parsl.dataflow.dflow DEBUG: Task 245 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 245 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 245 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 245 has memoization hash 4efea3eee8538f3d978d37dbedc66ffb parsl.dataflow.memoization INFO: Task 245 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e8793c4900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 245 try 0 launched on executor multi with executor id 246 parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/81f808ef-9ef5-4c75-9015-4da76eb1302c_isrForCrosstalkSources_1124087_27.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/81f808ef-9ef5-4c75-9015-4da76eb1302c_isrForCrosstalkSources_1124087_27.stderr parsl.dataflow.dflow DEBUG: Task 246 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 246 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 246 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 246 has memoization hash 9afa8adacf970c20b5d3d92f69a45cce parsl.dataflow.memoization INFO: Task 246 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e8793c7ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 246 try 0 launched on executor multi with executor id 247 parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/d3414de2-7a4d-45b0-8f2f-f82781a8c644_isrForCrosstalkSources_1123958_16.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/d3414de2-7a4d-45b0-8f2f-f82781a8c644_isrForCrosstalkSources_1123958_16.stderr parsl.dataflow.dflow DEBUG: Task 247 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 247 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 247 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 247 has memoization hash 11ee8f98c333320c9acf3d79532138c4 parsl.dataflow.memoization INFO: Task 247 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e8793c7e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 247 try 0 launched on executor multi with executor id 248 parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/8a73cfc3-7b7c-4e48-95d3-37e37d63d598_isrForCrosstalkSources_1123958_18.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/8a73cfc3-7b7c-4e48-95d3-37e37d63d598_isrForCrosstalkSources_1123958_18.stderr 2024-03-07 03:34:01 proc_lsst.multi:170 [INFO] [multi] job_id 16884174 proc_lsst.multi INFO: [multi] job_id 16884174 2024-03-07 03:34:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 03:34:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:01 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884174 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884174 parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 16884174 parsl.executors.status_handling INFO: Allocated block ID 4 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4 2024-03-07 03:34:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 2024-03-07 03:34:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 03:34:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:01 proc_lsst.multi:170 [INFO] [multi] job_id 16884175 proc_lsst.multi INFO: [multi] job_id 16884175 2024-03-07 03:34:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 03:34:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:01 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884175 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884175 parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 16884175 parsl.executors.status_handling INFO: Allocated block ID 5 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5 2024-03-07 03:34:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 2024-03-07 03:34:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 03:34:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:01 proc_lsst.multi:170 [INFO] [multi] job_id 16884176 proc_lsst.multi INFO: [multi] job_id 16884176 2024-03-07 03:34:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:34:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:01 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884176 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884176 parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 16884176 parsl.executors.status_handling INFO: Allocated block ID 6 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6 2024-03-07 03:34:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 2024-03-07 03:34:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:34:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:01 proc_lsst.multi:170 [INFO] [multi] job_id 16884177 proc_lsst.multi INFO: [multi] job_id 16884177 2024-03-07 03:34:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-07 03:34:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:01 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884177 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884177 parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 16884177 parsl.executors.status_handling INFO: Allocated block ID 7 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7 2024-03-07 03:34:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 2024-03-07 03:34:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-07 03:34:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:02 proc_lsst.multi:170 [INFO] [multi] job_id 16884178 proc_lsst.multi INFO: [multi] job_id 16884178 2024-03-07 03:34:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-07 03:34:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:02 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884178 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884178 parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 16884178 parsl.executors.status_handling INFO: Allocated block ID 8 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8 2024-03-07 03:34:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 2024-03-07 03:34:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-07 03:34:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:02 proc_lsst.multi:170 [INFO] [multi] job_id 16884179 proc_lsst.multi INFO: [multi] job_id 16884179 2024-03-07 03:34:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-07 03:34:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:02 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884179 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884179 parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 16884179 parsl.executors.status_handling INFO: Allocated block ID 9 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9 2024-03-07 03:34:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 2024-03-07 03:34:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-07 03:34:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:02 proc_lsst.multi:170 [INFO] [multi] job_id 16884180 proc_lsst.multi INFO: [multi] job_id 16884180 2024-03-07 03:34:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-07 03:34:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:02 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884180 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884180 parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 16884180 parsl.executors.status_handling INFO: Allocated block ID 10 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10 2024-03-07 03:34:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 2024-03-07 03:34:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-07 03:34:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:02 proc_lsst.multi:170 [INFO] [multi] job_id 16884181 proc_lsst.multi INFO: [multi] job_id 16884181 2024-03-07 03:34:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-07 03:34:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:02 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884181 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884181 parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 16884181 parsl.executors.status_handling INFO: Allocated block ID 11 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11 2024-03-07 03:34:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 2024-03-07 03:34:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-07 03:34:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:03 proc_lsst.multi:170 [INFO] [multi] job_id 16884182 proc_lsst.multi INFO: [multi] job_id 16884182 2024-03-07 03:34:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-07 03:34:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:03 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884182 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884182 parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 16884182 parsl.executors.status_handling INFO: Allocated block ID 12 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12 2024-03-07 03:34:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 2024-03-07 03:34:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-07 03:34:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:03 proc_lsst.multi:170 [INFO] [multi] job_id 16884183 proc_lsst.multi INFO: [multi] job_id 16884183 2024-03-07 03:34:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-07 03:34:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:03 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884183 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884183 parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 16884183 parsl.executors.status_handling INFO: Allocated block ID 13 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13 2024-03-07 03:34:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 2024-03-07 03:34:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-07 03:34:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:03 proc_lsst.multi:170 [INFO] [multi] job_id 16884184 proc_lsst.multi INFO: [multi] job_id 16884184 2024-03-07 03:34:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-07 03:34:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:03 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884184 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884184 parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 16884184 parsl.executors.status_handling INFO: Allocated block ID 14 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14 2024-03-07 03:34:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 2024-03-07 03:34:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-07 03:34:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:04 proc_lsst.multi:170 [INFO] [multi] job_id 16884185 proc_lsst.multi INFO: [multi] job_id 16884185 2024-03-07 03:34:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-07 03:34:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:04 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884185 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884185 parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 16884185 parsl.executors.status_handling INFO: Allocated block ID 15 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15 2024-03-07 03:34:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 2024-03-07 03:34:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-07 03:34:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:04 proc_lsst.multi:170 [INFO] [multi] job_id 16884186 proc_lsst.multi INFO: [multi] job_id 16884186 2024-03-07 03:34:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-07 03:34:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:04 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884186 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884186 parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 16884186 parsl.executors.status_handling INFO: Allocated block ID 16 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16 2024-03-07 03:34:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 2024-03-07 03:34:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-07 03:34:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:04 proc_lsst.multi:170 [INFO] [multi] job_id 16884187 proc_lsst.multi INFO: [multi] job_id 16884187 2024-03-07 03:34:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-07 03:34:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:04 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884187 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884187 parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 16884187 parsl.executors.status_handling INFO: Allocated block ID 17 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17 2024-03-07 03:34:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 2024-03-07 03:34:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-07 03:34:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:04 proc_lsst.multi:170 [INFO] [multi] job_id 16884188 proc_lsst.multi INFO: [multi] job_id 16884188 2024-03-07 03:34:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-07 03:34:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:04 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884188 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884188 parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 16884188 parsl.executors.status_handling INFO: Allocated block ID 18 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18 2024-03-07 03:34:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 2024-03-07 03:34:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-07 03:34:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:05 proc_lsst.multi:170 [INFO] [multi] job_id 16884189 proc_lsst.multi INFO: [multi] job_id 16884189 2024-03-07 03:34:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-07 03:34:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:05 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884189 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884189 parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 16884189 parsl.executors.status_handling INFO: Allocated block ID 19 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19 2024-03-07 03:34:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 2024-03-07 03:34:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-07 03:34:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:05 proc_lsst.multi:170 [INFO] [multi] job_id 16884190 proc_lsst.multi INFO: [multi] job_id 16884190 2024-03-07 03:34:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-07 03:34:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:05 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884190 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884190 parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 16884190 parsl.executors.status_handling INFO: Allocated block ID 20 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20 2024-03-07 03:34:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 2024-03-07 03:34:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-07 03:34:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:05 proc_lsst.multi:170 [INFO] [multi] job_id 16884191 proc_lsst.multi INFO: [multi] job_id 16884191 2024-03-07 03:34:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 03:34:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:05 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884191 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884191 parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 16884191 parsl.executors.status_handling INFO: Allocated block ID 21 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21 2024-03-07 03:34:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 2024-03-07 03:34:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 03:34:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:05 proc_lsst.multi:170 [INFO] [multi] job_id 16884192 proc_lsst.multi INFO: [multi] job_id 16884192 2024-03-07 03:34:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-07 03:34:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:05 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884192 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884192 parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 16884192 parsl.executors.status_handling INFO: Allocated block ID 22 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22 2024-03-07 03:34:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 2024-03-07 03:34:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-07 03:34:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:05 proc_lsst.multi:170 [INFO] [multi] job_id 16884193 proc_lsst.multi INFO: [multi] job_id 16884193 2024-03-07 03:34:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-07 03:34:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:05 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884193 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884193 parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 16884193 parsl.executors.status_handling INFO: Allocated block ID 23 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23 2024-03-07 03:34:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 2024-03-07 03:34:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-07 03:34:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:06 proc_lsst.multi:170 [INFO] [multi] job_id 16884194 proc_lsst.multi INFO: [multi] job_id 16884194 2024-03-07 03:34:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-07 03:34:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:06 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884194 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884194 parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 16884194 parsl.executors.status_handling INFO: Allocated block ID 24 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24 2024-03-07 03:34:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 2024-03-07 03:34:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-07 03:34:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:06 proc_lsst.multi:170 [INFO] [multi] job_id 16884195 proc_lsst.multi INFO: [multi] job_id 16884195 2024-03-07 03:34:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-07 03:34:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:06 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884195 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884195 parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 16884195 parsl.executors.status_handling INFO: Allocated block ID 25 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25 2024-03-07 03:34:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 2024-03-07 03:34:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-07 03:34:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:06 proc_lsst.multi:170 [INFO] [multi] job_id 16884196 proc_lsst.multi INFO: [multi] job_id 16884196 2024-03-07 03:34:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-07 03:34:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:06 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884196 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884196 parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 16884196 parsl.executors.status_handling INFO: Allocated block ID 26 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26 2024-03-07 03:34:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 2024-03-07 03:34:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-07 03:34:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:06 proc_lsst.multi:170 [INFO] [multi] job_id 16884197 proc_lsst.multi INFO: [multi] job_id 16884197 2024-03-07 03:34:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-07 03:34:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:06 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884197 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884197 parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 16884197 parsl.executors.status_handling INFO: Allocated block ID 27 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27 2024-03-07 03:34:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 2024-03-07 03:34:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-07 03:34:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:08 proc_lsst.multi:170 [INFO] [multi] job_id 16884203 proc_lsst.multi INFO: [multi] job_id 16884203 2024-03-07 03:34:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-07 03:34:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:08 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884203 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884203 parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 16884203 parsl.executors.status_handling INFO: Allocated block ID 28 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28 2024-03-07 03:34:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 2024-03-07 03:34:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-07 03:34:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:08 proc_lsst.multi:170 [INFO] [multi] job_id 16884204 proc_lsst.multi INFO: [multi] job_id 16884204 2024-03-07 03:34:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-07 03:34:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:08 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884204 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884204 parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 16884204 parsl.executors.status_handling INFO: Allocated block ID 29 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29 2024-03-07 03:34:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 2024-03-07 03:34:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-07 03:34:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:09 proc_lsst.multi:170 [INFO] [multi] job_id 16884205 proc_lsst.multi INFO: [multi] job_id 16884205 2024-03-07 03:34:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-07 03:34:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:09 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884205 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884205 parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 16884205 parsl.executors.status_handling INFO: Allocated block ID 30 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30 2024-03-07 03:34:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 2024-03-07 03:34:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-07 03:34:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:09 proc_lsst.multi:170 [INFO] [multi] job_id 16884206 proc_lsst.multi INFO: [multi] job_id 16884206 2024-03-07 03:34:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:34:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:09 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884206 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884206 parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 16884206 parsl.executors.status_handling INFO: Allocated block ID 31 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-31 2024-03-07 03:34:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 2024-03-07 03:34:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:34:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:09 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-07 03:34:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:34:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:09 proc_lsst.multi:170 [INFO] [multi] job_id 16884207 proc_lsst.multi INFO: [multi] job_id 16884207 2024-03-07 03:34:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:34:09 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16884207 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16884207 parsl.executors.status_handling DEBUG: Launched block 31 on executor multi with job ID 16884207 parsl.executors.status_handling INFO: Allocated block ID 32 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-32 2024-03-07 03:34:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 2024-03-07 03:34:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:34:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:09 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:34:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:09 proc_lsst.multi:170 [INFO] [multi] job_id 16884208 proc_lsst.multi INFO: [multi] job_id 16884208 2024-03-07 03:34:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 03:34:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:34:09 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16884208 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16884208 parsl.executors.status_handling DEBUG: Launched block 32 on executor multi with job ID 16884208 parsl.executors.status_handling INFO: Allocated block ID 33 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-33 2024-03-07 03:34:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 2024-03-07 03:34:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:34:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:09 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 03:34:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:34:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:09 proc_lsst.multi:170 [INFO] [multi] job_id 16884209 proc_lsst.multi INFO: [multi] job_id 16884209 2024-03-07 03:34:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 03:34:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:34:09 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16884209 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16884209 parsl.executors.status_handling DEBUG: Launched block 33 on executor multi with job ID 16884209 parsl.executors.status_handling INFO: Allocated block ID 34 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-34 2024-03-07 03:34:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 2024-03-07 03:34:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:34:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:09 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 03:34:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:34:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:10 proc_lsst.multi:170 [INFO] [multi] job_id 16884210 proc_lsst.multi INFO: [multi] job_id 16884210 2024-03-07 03:34:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 03:34:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:34:10 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16884210 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16884210 parsl.executors.status_handling DEBUG: Launched block 34 on executor multi with job ID 16884210 parsl.executors.status_handling INFO: Allocated block ID 35 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-35 2024-03-07 03:34:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 2024-03-07 03:34:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:34:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:10 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 03:34:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:34:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5162d17d39101d83a47b696f7604c1a5 with result from task 0 parsl.dataflow.dflow DEBUG: Task 0 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/b0e6bdf0-d9d7-4b0d-b39a-56f000229767_isrForCrosstalkSources_1123954_15.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/b0e6bdf0-d9d7-4b0d-b39a-56f000229767_isrForCrosstalkSources_1123954_15.stderr 2024-03-07 03:34:10 proc_lsst.multi:170 [INFO] [multi] job_id 16884211 proc_lsst.multi INFO: [multi] job_id 16884211 2024-03-07 03:34:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:34:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:34:10 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16884211 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16884211 parsl.executors.status_handling DEBUG: Launched block 35 on executor multi with job ID 16884211 parsl.executors.status_handling INFO: Allocated block ID 36 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-36 2024-03-07 03:34:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 2024-03-07 03:34:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:34:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:10 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:34:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:34:10 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-07 03:34:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 660bde7bf87994c1a7b5454bd3fd08c1 with result from task 1 parsl.dataflow.dflow DEBUG: Task 1 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/5a11f9c0-d6e9-47ff-a71f-8daa9cefdbed_isrForCrosstalkSources_1123956_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/5a11f9c0-d6e9-47ff-a71f-8daa9cefdbed_isrForCrosstalkSources_1123956_47.stderr 2024-03-07 03:34:10 proc_lsst.multi:170 [INFO] [multi] job_id 16884212 proc_lsst.multi INFO: [multi] job_id 16884212 2024-03-07 03:34:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:10 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16884212 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16884212 parsl.executors.status_handling DEBUG: Launched block 36 on executor multi with job ID 16884212 parsl.executors.status_handling INFO: Allocated block ID 37 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-37 2024-03-07 03:34:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 2024-03-07 03:34:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:34:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:10 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:34:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:34:10 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:10 proc_lsst.multi:170 [INFO] [multi] job_id 16884213 proc_lsst.multi INFO: [multi] job_id 16884213 2024-03-07 03:34:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 03:34:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:10 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16884213 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16884213 parsl.executors.status_handling DEBUG: Launched block 37 on executor multi with job ID 16884213 parsl.executors.status_handling INFO: Allocated block ID 38 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-38 2024-03-07 03:34:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 2024-03-07 03:34:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:34:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:10 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:34:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:34:10 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 03:34:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:10 proc_lsst.multi:170 [INFO] [multi] job_id 16884214 proc_lsst.multi INFO: [multi] job_id 16884214 2024-03-07 03:34:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 03:34:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:10 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16884214 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16884214 parsl.executors.status_handling DEBUG: Launched block 38 on executor multi with job ID 16884214 parsl.executors.status_handling INFO: Allocated block ID 39 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-39 2024-03-07 03:34:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 2024-03-07 03:34:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:34:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:10 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:34:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:34:10 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 03:34:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:11 proc_lsst.multi:170 [INFO] [multi] job_id 16884215 proc_lsst.multi INFO: [multi] job_id 16884215 2024-03-07 03:34:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 03:34:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:11 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16884215 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16884215 parsl.executors.status_handling DEBUG: Launched block 39 on executor multi with job ID 16884215 parsl.executors.status_handling INFO: Allocated block ID 40 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-40 2024-03-07 03:34:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 2024-03-07 03:34:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:34:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:11 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:34:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:34:11 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 03:34:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:11 proc_lsst.multi:170 [INFO] [multi] job_id 16884216 proc_lsst.multi INFO: [multi] job_id 16884216 2024-03-07 03:34:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:34:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:11 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16884216 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16884216 parsl.executors.status_handling DEBUG: Launched block 40 on executor multi with job ID 16884216 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22988724935824 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 41, active_tasks = 246 parsl.dataflow.strategy DEBUG: Executor multi has 246 active tasks, 1/40 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 41 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 21 more blocks parsl.executors.status_handling INFO: Scaling out by 21 blocks parsl.executors.status_handling INFO: Allocated block ID 41 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-41 2024-03-07 03:34:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 2024-03-07 03:34:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:34:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:11 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:34:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:34:11 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:34:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:11 proc_lsst.multi:170 [INFO] [multi] job_id 16884217 proc_lsst.multi INFO: [multi] job_id 16884217 2024-03-07 03:34:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-07 03:34:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:11 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16884217 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16884217 parsl.executors.status_handling DEBUG: Launched block 41 on executor multi with job ID 16884217 parsl.executors.status_handling INFO: Allocated block ID 42 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-42 2024-03-07 03:34:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 2024-03-07 03:34:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:34:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:11 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:34:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:34:11 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-07 03:34:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:11 proc_lsst.multi:170 [INFO] [multi] job_id 16884218 proc_lsst.multi INFO: [multi] job_id 16884218 2024-03-07 03:34:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-07 03:34:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:11 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16884218 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16884218 parsl.executors.status_handling DEBUG: Launched block 42 on executor multi with job ID 16884218 parsl.executors.status_handling INFO: Allocated block ID 43 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-43 2024-03-07 03:34:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 2024-03-07 03:34:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:34:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:11 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:34:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:34:11 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-07 03:34:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:12 proc_lsst.multi:170 [INFO] [multi] job_id 16884219 proc_lsst.multi INFO: [multi] job_id 16884219 2024-03-07 03:34:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-07 03:34:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:12 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16884219 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16884219 parsl.executors.status_handling DEBUG: Launched block 43 on executor multi with job ID 16884219 parsl.executors.status_handling INFO: Allocated block ID 44 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-44 2024-03-07 03:34:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 2024-03-07 03:34:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:34:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:12 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:34:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:34:12 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-07 03:34:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:12 proc_lsst.multi:170 [INFO] [multi] job_id 16884220 proc_lsst.multi INFO: [multi] job_id 16884220 2024-03-07 03:34:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-07 03:34:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:12 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16884220 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16884220 parsl.executors.status_handling DEBUG: Launched block 44 on executor multi with job ID 16884220 parsl.executors.status_handling INFO: Allocated block ID 45 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-45 2024-03-07 03:34:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 2024-03-07 03:34:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:34:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:12 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:34:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:34:12 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-07 03:34:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:12 proc_lsst.multi:170 [INFO] [multi] job_id 16884221 proc_lsst.multi INFO: [multi] job_id 16884221 2024-03-07 03:34:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-07 03:34:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:12 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16884221 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16884221 parsl.executors.status_handling DEBUG: Launched block 45 on executor multi with job ID 16884221 parsl.executors.status_handling INFO: Allocated block ID 46 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-46 2024-03-07 03:34:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 2024-03-07 03:34:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:34:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:12 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:34:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:34:12 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-07 03:34:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:12 proc_lsst.multi:170 [INFO] [multi] job_id 16884222 proc_lsst.multi INFO: [multi] job_id 16884222 2024-03-07 03:34:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-07 03:34:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:12 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16884222 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16884222 parsl.executors.status_handling DEBUG: Launched block 46 on executor multi with job ID 16884222 parsl.executors.status_handling INFO: Allocated block ID 47 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-47 2024-03-07 03:34:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 2024-03-07 03:34:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:34:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:12 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:34:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:34:12 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-07 03:34:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:13 proc_lsst.multi:170 [INFO] [multi] job_id 16884223 proc_lsst.multi INFO: [multi] job_id 16884223 2024-03-07 03:34:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-07 03:34:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:13 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16884223 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16884223 parsl.executors.status_handling DEBUG: Launched block 47 on executor multi with job ID 16884223 parsl.executors.status_handling INFO: Allocated block ID 48 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-48 2024-03-07 03:34:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 2024-03-07 03:34:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:34:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:13 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:34:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:34:13 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-07 03:34:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:13 proc_lsst.multi:170 [INFO] [multi] job_id 16884224 proc_lsst.multi INFO: [multi] job_id 16884224 2024-03-07 03:34:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-07 03:34:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:13 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16884224 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16884224 parsl.executors.status_handling DEBUG: Launched block 48 on executor multi with job ID 16884224 parsl.executors.status_handling INFO: Allocated block ID 49 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-49 2024-03-07 03:34:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 2024-03-07 03:34:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:34:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:13 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:34:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:34:13 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-07 03:34:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:13 proc_lsst.multi:170 [INFO] [multi] job_id 16884225 proc_lsst.multi INFO: [multi] job_id 16884225 2024-03-07 03:34:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-07 03:34:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:13 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16884225 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16884225 parsl.executors.status_handling DEBUG: Launched block 49 on executor multi with job ID 16884225 parsl.executors.status_handling INFO: Allocated block ID 50 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-50 2024-03-07 03:34:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 2024-03-07 03:34:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:34:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:13 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:34:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:34:13 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-07 03:34:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:13 proc_lsst.multi:170 [INFO] [multi] job_id 16884226 proc_lsst.multi INFO: [multi] job_id 16884226 2024-03-07 03:34:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-07 03:34:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:13 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16884226 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16884226 parsl.executors.status_handling DEBUG: Launched block 50 on executor multi with job ID 16884226 parsl.executors.status_handling INFO: Allocated block ID 51 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-51 2024-03-07 03:34:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 2024-03-07 03:34:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:34:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:13 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:34:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:34:13 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-07 03:34:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:14 proc_lsst.multi:170 [INFO] [multi] job_id 16884227 proc_lsst.multi INFO: [multi] job_id 16884227 2024-03-07 03:34:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-07 03:34:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:14 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16884227 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16884227 parsl.executors.status_handling DEBUG: Launched block 51 on executor multi with job ID 16884227 parsl.executors.status_handling INFO: Allocated block ID 52 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-52 2024-03-07 03:34:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 2024-03-07 03:34:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:34:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:14 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:34:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:34:14 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-07 03:34:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:14 proc_lsst.multi:170 [INFO] [multi] job_id 16884228 proc_lsst.multi INFO: [multi] job_id 16884228 2024-03-07 03:34:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-07 03:34:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:14 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16884228 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16884228 parsl.executors.status_handling DEBUG: Launched block 52 on executor multi with job ID 16884228 parsl.executors.status_handling INFO: Allocated block ID 53 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-53 2024-03-07 03:34:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 2024-03-07 03:34:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:34:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:14 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:34:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:34:14 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-07 03:34:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:14 proc_lsst.multi:170 [INFO] [multi] job_id 16884229 proc_lsst.multi INFO: [multi] job_id 16884229 2024-03-07 03:34:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-07 03:34:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:14 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16884229 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16884229 parsl.executors.status_handling DEBUG: Launched block 53 on executor multi with job ID 16884229 parsl.executors.status_handling INFO: Allocated block ID 54 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-54 2024-03-07 03:34:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 2024-03-07 03:34:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:34:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:14 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:34:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:34:14 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-07 03:34:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:14 proc_lsst.multi:170 [INFO] [multi] job_id 16884230 proc_lsst.multi INFO: [multi] job_id 16884230 2024-03-07 03:34:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-07 03:34:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:14 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16884230 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16884230 parsl.executors.status_handling DEBUG: Launched block 54 on executor multi with job ID 16884230 parsl.executors.status_handling INFO: Allocated block ID 55 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-55 2024-03-07 03:34:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 2024-03-07 03:34:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:34:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:14 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:34:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:34:14 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-07 03:34:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:14 proc_lsst.multi:170 [INFO] [multi] job_id 16884231 proc_lsst.multi INFO: [multi] job_id 16884231 2024-03-07 03:34:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 03:34:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:14 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16884231 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16884231 parsl.executors.status_handling DEBUG: Launched block 55 on executor multi with job ID 16884231 parsl.executors.status_handling INFO: Allocated block ID 56 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-56 2024-03-07 03:34:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 2024-03-07 03:34:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:34:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:14 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:34:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:34:14 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 03:34:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:14 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-07 03:34:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 03:34:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:15 proc_lsst.multi:170 [INFO] [multi] job_id 16884232 proc_lsst.multi INFO: [multi] job_id 16884232 2024-03-07 03:34:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 03:34:15 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16884232 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16884232 parsl.executors.status_handling DEBUG: Launched block 56 on executor multi with job ID 16884232 parsl.executors.status_handling INFO: Allocated block ID 57 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-57 2024-03-07 03:34:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 2024-03-07 03:34:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:34:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:34:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:34:15 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 03:34:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:15 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 03:34:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:15 proc_lsst.multi:170 [INFO] [multi] job_id 16884233 proc_lsst.multi INFO: [multi] job_id 16884233 2024-03-07 03:34:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 03:34:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 03:34:15 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16884233 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16884233 parsl.executors.status_handling DEBUG: Launched block 57 on executor multi with job ID 16884233 parsl.executors.status_handling INFO: Allocated block ID 58 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-58 2024-03-07 03:34:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 2024-03-07 03:34:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:34:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:34:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:34:15 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 03:34:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:15 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 03:34:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 03:34:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:15 proc_lsst.multi:170 [INFO] [multi] job_id 16884234 proc_lsst.multi INFO: [multi] job_id 16884234 2024-03-07 03:34:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 03:34:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 03:34:15 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16884234 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16884234 parsl.executors.status_handling DEBUG: Launched block 58 on executor multi with job ID 16884234 parsl.executors.status_handling INFO: Allocated block ID 59 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-59 2024-03-07 03:34:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 2024-03-07 03:34:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:34:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:34:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:34:15 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 03:34:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:15 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 03:34:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 03:34:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:15 proc_lsst.multi:170 [INFO] [multi] job_id 16884235 proc_lsst.multi INFO: [multi] job_id 16884235 2024-03-07 03:34:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 03:34:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 03:34:15 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16884235 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16884235 parsl.executors.status_handling DEBUG: Launched block 59 on executor multi with job ID 16884235 parsl.executors.status_handling INFO: Allocated block ID 60 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-60 2024-03-07 03:34:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 2024-03-07 03:34:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:34:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:34:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:34:15 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 03:34:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:15 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 03:34:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 03:34:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:15 proc_lsst.multi:170 [INFO] [multi] job_id 16884236 proc_lsst.multi INFO: [multi] job_id 16884236 2024-03-07 03:34:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:34:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 03:34:15 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16884236 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16884236 parsl.executors.status_handling DEBUG: Launched block 60 on executor multi with job ID 16884236 parsl.executors.status_handling INFO: Allocated block ID 61 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-61 2024-03-07 03:34:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 2024-03-07 03:34:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:34:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:34:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:34:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:34:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:34:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:34:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:34:15 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 03:34:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:34:15 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:34:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:34:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 03:34:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54508 --result_port=54825 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:34:15 proc_lsst.multi:170 [INFO] [multi] job_id 16884237 proc_lsst.multi INFO: [multi] job_id 16884237 2024-03-07 03:34:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-07 03:34:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 03:34:15 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16884237 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16884237 parsl.executors.status_handling DEBUG: Launched block 61 on executor multi with job ID 16884237 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22988724935824 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 246 parsl.dataflow.strategy DEBUG: Executor multi has 246 active tasks, 1/61 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22988724935824 parsl.dataflow.dflow INFO: Task 2 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1dd3f8228bb419af7907dcfb9ab8c113 with result from task 2 parsl.dataflow.dflow DEBUG: Task 2 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/ad5ad05d-a107-41d4-bc57-cb9ae32c4931_isrForCrosstalkSources_1123956_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/ad5ad05d-a107-41d4-bc57-cb9ae32c4931_isrForCrosstalkSources_1123956_50.stderr parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7d023f7829da0c7125aa11d06debaf3 with result from task 3 parsl.dataflow.dflow DEBUG: Task 3 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/927f30b4-db8e-475f-aeb5-81b93f9594de_isrForCrosstalkSources_1124087_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/927f30b4-db8e-475f-aeb5-81b93f9594de_isrForCrosstalkSources_1124087_42.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 244 parsl.dataflow.strategy DEBUG: Executor multi has 244 active tasks, 1/61 running/pending blocks, and 32 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22988724935824 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 244 parsl.dataflow.strategy DEBUG: Executor multi has 244 active tasks, 1/61 running/pending blocks, and 32 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22988724935824 parsl.dataflow.dflow INFO: Task 34 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 231e6d95f629e6ee2c87720495cfd50c with result from task 34 parsl.dataflow.dflow DEBUG: Task 34 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/1f226936-3ce9-4dab-b8cb-d2d677d766d3_isrForCrosstalkSources_1123958_30.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/1f226936-3ce9-4dab-b8cb-d2d677d766d3_isrForCrosstalkSources_1123958_30.stderr parsl.dataflow.dflow INFO: Task 35 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9aaa72de14162f50c8e4729b00165009 with result from task 35 parsl.dataflow.dflow DEBUG: Task 35 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/07c97ef6-8e48-4554-a8b5-b1a3b214fb92_isrForCrosstalkSources_1123958_46.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/07c97ef6-8e48-4554-a8b5-b1a3b214fb92_isrForCrosstalkSources_1123958_46.stderr 2024-03-07 03:34:31 proc_lsst.multi:146 [INFO] found job 766 in provider local proc_lsst.multi INFO: found job 766 in provider local 2024-03-07 03:34:31 proc_lsst.multi:146 [INFO] found job 16884172 in provider astro proc_lsst.multi INFO: found job 16884172 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:31 proc_lsst.multi:146 [INFO] found job 16884173 in provider astro proc_lsst.multi INFO: found job 16884173 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:32 proc_lsst.multi:146 [INFO] found job 16884174 in provider astro proc_lsst.multi INFO: found job 16884174 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:32 proc_lsst.multi:146 [INFO] found job 16884175 in provider astro proc_lsst.multi INFO: found job 16884175 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:33 proc_lsst.multi:146 [INFO] found job 16884176 in provider astro proc_lsst.multi INFO: found job 16884176 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:33 proc_lsst.multi:146 [INFO] found job 16884177 in provider astro proc_lsst.multi INFO: found job 16884177 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:34 proc_lsst.multi:146 [INFO] found job 16884178 in provider astro proc_lsst.multi INFO: found job 16884178 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:34 proc_lsst.multi:146 [INFO] found job 16884179 in provider astro proc_lsst.multi INFO: found job 16884179 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 227060416eb2d0fc5ff2112335d0c5ad with result from task 36 parsl.dataflow.dflow DEBUG: Task 36 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/e0f29fca-715c-4814-84e2-bfadc5f1d95e_isrForCrosstalkSources_1123956_5.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/e0f29fca-715c-4814-84e2-bfadc5f1d95e_isrForCrosstalkSources_1123956_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:34 proc_lsst.multi:146 [INFO] found job 16884180 in provider astro proc_lsst.multi INFO: found job 16884180 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:35 proc_lsst.multi:146 [INFO] found job 16884181 in provider astro proc_lsst.multi INFO: found job 16884181 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:35 proc_lsst.multi:146 [INFO] found job 16884182 in provider astro proc_lsst.multi INFO: found job 16884182 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:35 proc_lsst.multi:146 [INFO] found job 16884183 in provider astro proc_lsst.multi INFO: found job 16884183 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:35 proc_lsst.multi:146 [INFO] found job 16884184 in provider astro proc_lsst.multi INFO: found job 16884184 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.dataflow.dflow INFO: Task 37 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0480b53c0b389b785cf8fcdd91b6cbf3 with result from task 37 parsl.dataflow.dflow DEBUG: Task 37 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/9a355d03-dd39-4d87-88a8-d07a468bc9d7_isrForCrosstalkSources_1123956_33.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/9a355d03-dd39-4d87-88a8-d07a468bc9d7_isrForCrosstalkSources_1123956_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:36 proc_lsst.multi:146 [INFO] found job 16884185 in provider astro proc_lsst.multi INFO: found job 16884185 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:36 proc_lsst.multi:146 [INFO] found job 16884186 in provider astro proc_lsst.multi INFO: found job 16884186 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:36 proc_lsst.multi:146 [INFO] found job 16884187 in provider astro proc_lsst.multi INFO: found job 16884187 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:37 proc_lsst.multi:146 [INFO] found job 16884188 in provider astro proc_lsst.multi INFO: found job 16884188 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:37 proc_lsst.multi:146 [INFO] found job 16884189 in provider astro proc_lsst.multi INFO: found job 16884189 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:37 proc_lsst.multi:146 [INFO] found job 16884190 in provider astro proc_lsst.multi INFO: found job 16884190 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:38 proc_lsst.multi:146 [INFO] found job 16884191 in provider astro proc_lsst.multi INFO: found job 16884191 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:38 proc_lsst.multi:146 [INFO] found job 16884192 in provider astro proc_lsst.multi INFO: found job 16884192 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:40 proc_lsst.multi:146 [INFO] found job 16884193 in provider astro proc_lsst.multi INFO: found job 16884193 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:40 proc_lsst.multi:146 [INFO] found job 16884194 in provider astro proc_lsst.multi INFO: found job 16884194 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:41 proc_lsst.multi:146 [INFO] found job 16884195 in provider astro proc_lsst.multi INFO: found job 16884195 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:41 proc_lsst.multi:146 [INFO] found job 16884196 in provider astro proc_lsst.multi INFO: found job 16884196 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:41 proc_lsst.multi:146 [INFO] found job 16884197 in provider astro proc_lsst.multi INFO: found job 16884197 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:42 proc_lsst.multi:146 [INFO] found job 16884203 in provider astro proc_lsst.multi INFO: found job 16884203 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:42 proc_lsst.multi:146 [INFO] found job 16884204 in provider astro proc_lsst.multi INFO: found job 16884204 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:42 proc_lsst.multi:146 [INFO] found job 16884205 in provider astro proc_lsst.multi INFO: found job 16884205 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:43 proc_lsst.multi:146 [INFO] found job 16884206 in provider astro proc_lsst.multi INFO: found job 16884206 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:43 proc_lsst.multi:146 [INFO] found job 16884207 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884207 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884207,16884208,16884209,16884210,16884211' parsl.dataflow.dflow INFO: Task 58 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 606a9a43fa8b6f298d4707551f339a11 with result from task 58 parsl.dataflow.dflow DEBUG: Task 58 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/58622b96-908f-49bd-a6f2-fd25ae00cc6d_isrForCrosstalkSources_1123956_30.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/58622b96-908f-49bd-a6f2-fd25ae00cc6d_isrForCrosstalkSources_1123956_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884207 R 16884208 R 16884209 R 16884210 R 16884211 R parsl.providers.slurm.slurm DEBUG: Updating job 16884207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884211 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:43 proc_lsst.multi:146 [INFO] found job 16884208 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884208 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884207,16884208,16884209,16884210,16884211' parsl.providers.slurm.slurm DEBUG: squeue returned 16884207 R 16884208 R 16884209 R 16884210 R 16884211 R parsl.providers.slurm.slurm DEBUG: Updating job 16884207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884211 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:43 proc_lsst.multi:146 [INFO] found job 16884209 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884209 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884207,16884208,16884209,16884210,16884211' parsl.providers.slurm.slurm DEBUG: squeue returned 16884207 R 16884208 R 16884209 R 16884210 R 16884211 R parsl.providers.slurm.slurm DEBUG: Updating job 16884207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884211 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:44 proc_lsst.multi:146 [INFO] found job 16884210 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884210 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884207,16884208,16884209,16884210,16884211' parsl.providers.slurm.slurm DEBUG: squeue returned 16884207 R 16884208 R 16884209 R 16884210 R 16884211 R parsl.providers.slurm.slurm DEBUG: Updating job 16884207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884211 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:44 proc_lsst.multi:146 [INFO] found job 16884211 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884211 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884207,16884208,16884209,16884210,16884211' parsl.dataflow.dflow INFO: Task 64 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb91a40120038786c0bbe0861c066ca5 with result from task 64 parsl.dataflow.dflow DEBUG: Task 64 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/0b5ca7dc-7dcf-466d-af7a-a2f0bcdb7f0c_isrForCrosstalkSources_1123956_25.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/0b5ca7dc-7dcf-466d-af7a-a2f0bcdb7f0c_isrForCrosstalkSources_1123956_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884207 R 16884208 R 16884209 R 16884210 R 16884211 R parsl.providers.slurm.slurm DEBUG: Updating job 16884207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884211 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:44 proc_lsst.multi:146 [INFO] found job 16884212 in provider escience_a40 proc_lsst.multi INFO: found job 16884212 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:44 proc_lsst.multi:146 [INFO] found job 16884213 in provider escience_a40 proc_lsst.multi INFO: found job 16884213 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:45 proc_lsst.multi:146 [INFO] found job 16884214 in provider escience_a40 proc_lsst.multi INFO: found job 16884214 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:45 proc_lsst.multi:146 [INFO] found job 16884215 in provider escience_a40 proc_lsst.multi INFO: found job 16884215 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:45 proc_lsst.multi:146 [INFO] found job 16884216 in provider escience_a40 proc_lsst.multi INFO: found job 16884216 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:46 proc_lsst.multi:146 [INFO] found job 16884217 in provider escience_a40 proc_lsst.multi INFO: found job 16884217 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:46 proc_lsst.multi:146 [INFO] found job 16884218 in provider escience_a40 proc_lsst.multi INFO: found job 16884218 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:46 proc_lsst.multi:146 [INFO] found job 16884219 in provider escience_a40 proc_lsst.multi INFO: found job 16884219 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:47 proc_lsst.multi:146 [INFO] found job 16884220 in provider escience_a40 proc_lsst.multi INFO: found job 16884220 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:47 proc_lsst.multi:146 [INFO] found job 16884221 in provider escience_a40 proc_lsst.multi INFO: found job 16884221 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:47 proc_lsst.multi:146 [INFO] found job 16884222 in provider escience_a40 proc_lsst.multi INFO: found job 16884222 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:48 proc_lsst.multi:146 [INFO] found job 16884223 in provider escience_a40 proc_lsst.multi INFO: found job 16884223 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:48 proc_lsst.multi:146 [INFO] found job 16884224 in provider escience_a40 proc_lsst.multi INFO: found job 16884224 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:48 proc_lsst.multi:146 [INFO] found job 16884225 in provider escience_a40 proc_lsst.multi INFO: found job 16884225 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:49 proc_lsst.multi:146 [INFO] found job 16884226 in provider escience_a40 proc_lsst.multi INFO: found job 16884226 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:49 proc_lsst.multi:146 [INFO] found job 16884227 in provider escience_a40 proc_lsst.multi INFO: found job 16884227 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:49 proc_lsst.multi:146 [INFO] found job 16884228 in provider escience_a40 proc_lsst.multi INFO: found job 16884228 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:50 proc_lsst.multi:146 [INFO] found job 16884229 in provider escience_a40 proc_lsst.multi INFO: found job 16884229 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:50 proc_lsst.multi:146 [INFO] found job 16884230 in provider escience_a40 proc_lsst.multi INFO: found job 16884230 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:50 proc_lsst.multi:146 [INFO] found job 16884231 in provider escience_a40 proc_lsst.multi INFO: found job 16884231 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:51 proc_lsst.multi:146 [INFO] found job 16884232 in provider checkpoint proc_lsst.multi INFO: found job 16884232 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884232,16884233,16884234,16884235,16884236,16884237' parsl.providers.slurm.slurm DEBUG: squeue returned 16884237 PD 16884232 R 16884233 R 16884234 R 16884235 R 16884236 R parsl.providers.slurm.slurm DEBUG: Updating job 16884237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16884232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884236 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:51 proc_lsst.multi:146 [INFO] found job 16884233 in provider checkpoint proc_lsst.multi INFO: found job 16884233 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884232,16884233,16884234,16884235,16884236,16884237' parsl.providers.slurm.slurm DEBUG: squeue returned 16884237 PD 16884232 R 16884233 R 16884234 R 16884235 R 16884236 R parsl.providers.slurm.slurm DEBUG: Updating job 16884237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16884232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884236 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:51 proc_lsst.multi:146 [INFO] found job 16884234 in provider checkpoint proc_lsst.multi INFO: found job 16884234 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884232,16884233,16884234,16884235,16884236,16884237' parsl.providers.slurm.slurm DEBUG: squeue returned 16884237 PD 16884232 R 16884233 R 16884234 R 16884235 R 16884236 R parsl.providers.slurm.slurm DEBUG: Updating job 16884237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16884232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884236 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:52 proc_lsst.multi:146 [INFO] found job 16884235 in provider checkpoint proc_lsst.multi INFO: found job 16884235 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884232,16884233,16884234,16884235,16884236,16884237' parsl.providers.slurm.slurm DEBUG: squeue returned 16884237 PD 16884232 R 16884233 R 16884234 R 16884235 R 16884236 R parsl.providers.slurm.slurm DEBUG: Updating job 16884237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16884232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884236 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:52 proc_lsst.multi:146 [INFO] found job 16884236 in provider checkpoint proc_lsst.multi INFO: found job 16884236 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884232,16884233,16884234,16884235,16884236,16884237' parsl.dataflow.dflow INFO: Task 65 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3cd4ec5d53756be22f40eee186e2595f with result from task 65 parsl.dataflow.dflow DEBUG: Task 65 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/a8c169bc-ff2c-416c-be79-9fdeb5f3f8f2_isrForCrosstalkSources_1123958_23.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/a8c169bc-ff2c-416c-be79-9fdeb5f3f8f2_isrForCrosstalkSources_1123958_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884237 PD 16884232 R 16884233 R 16884234 R 16884235 R 16884236 R parsl.providers.slurm.slurm DEBUG: Updating job 16884237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16884232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884236 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:34:52 proc_lsst.multi:146 [INFO] found job 16884237 in provider checkpoint proc_lsst.multi INFO: found job 16884237 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884232,16884233,16884234,16884235,16884236,16884237' parsl.providers.slurm.slurm DEBUG: squeue returned 16884237 PD 16884232 R 16884233 R 16884234 R 16884235 R 16884236 R parsl.providers.slurm.slurm DEBUG: Updating job 16884237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16884232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884236 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 237 parsl.dataflow.strategy DEBUG: Executor multi has 237 active tasks, 61/1 running/pending blocks, and 65 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22988724935824 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 237 parsl.dataflow.strategy DEBUG: Executor multi has 237 active tasks, 61/1 running/pending blocks, and 65 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22988724935824 parsl.dataflow.dflow INFO: Task 66 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3775abce6c1fe606140894c6dbcaf41 with result from task 66 parsl.dataflow.dflow DEBUG: Task 66 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/ea21f981-0fb2-4527-9c9c-a2f39fb3ef0d_isrForCrosstalkSources_1123956_44.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/ea21f981-0fb2-4527-9c9c-a2f39fb3ef0d_isrForCrosstalkSources_1123956_44.stderr parsl.dataflow.dflow INFO: Task 7 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61cff9aa79cc7e795d89205eccb57de9 with result from task 7 parsl.dataflow.dflow DEBUG: Task 7 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/436af49a-9ebb-4618-ab8b-0d9f24a1df38_isrForCrosstalkSources_1123956_26.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/436af49a-9ebb-4618-ab8b-0d9f24a1df38_isrForCrosstalkSources_1123956_26.stderr parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 921fc569894e5d47bb18eaa41b1900a7 with result from task 9 parsl.dataflow.dflow DEBUG: Task 9 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/eb8b6fa9-29f2-421c-be14-7b07844bd0ee_isrForCrosstalkSources_1123958_4.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/eb8b6fa9-29f2-421c-be14-7b07844bd0ee_isrForCrosstalkSources_1123958_4.stderr parsl.dataflow.dflow INFO: Task 29 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f18974b03f8436c24ec6d0b462c69fcb with result from task 29 parsl.dataflow.dflow DEBUG: Task 29 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/574a538a-cbdc-46e6-98de-10af6e7194db_isrForCrosstalkSources_1123954_11.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/574a538a-cbdc-46e6-98de-10af6e7194db_isrForCrosstalkSources_1123954_11.stderr parsl.dataflow.dflow INFO: Task 20 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7da70dd9cd1ead8fdc175f4bfd5f5d4d with result from task 20 parsl.dataflow.dflow DEBUG: Task 20 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/f2b42fee-bed3-48d6-a018-551f29c2d92a_isrForCrosstalkSources_1123956_38.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/f2b42fee-bed3-48d6-a018-551f29c2d92a_isrForCrosstalkSources_1123956_38.stderr parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b9c469c3c6e8afd4934514c1bc339a1 with result from task 12 parsl.dataflow.dflow DEBUG: Task 12 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/b0cdd2b5-0061-4554-bb3c-12e251d38941_isrForCrosstalkSources_1123956_62.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/b0cdd2b5-0061-4554-bb3c-12e251d38941_isrForCrosstalkSources_1123956_62.stderr parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4681c80abb188ba517b88b38b2a764a with result from task 15 parsl.dataflow.dflow DEBUG: Task 15 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/f11dd8be-a266-4b2b-80c0-c049b8be4078_isrForCrosstalkSources_1123958_45.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/f11dd8be-a266-4b2b-80c0-c049b8be4078_isrForCrosstalkSources_1123958_45.stderr parsl.dataflow.dflow INFO: Task 8 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48ac6f7050a4f9858ac25ca258cca7c7 with result from task 8 parsl.dataflow.dflow DEBUG: Task 8 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/891a6879-75a1-4415-946b-a3b67ae88d3f_isrForCrosstalkSources_1123954_36.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/891a6879-75a1-4415-946b-a3b67ae88d3f_isrForCrosstalkSources_1123954_36.stderr parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72c9591e84f0e6614fb1b29f0ac4d92b with result from task 6 parsl.dataflow.dflow DEBUG: Task 6 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/bcc36014-3fd4-4300-b8f2-6a33823c01d9_isrForCrosstalkSources_1124087_25.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/bcc36014-3fd4-4300-b8f2-6a33823c01d9_isrForCrosstalkSources_1124087_25.stderr parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4defc78a8b2866da808743470ab9bd46 with result from task 30 parsl.dataflow.dflow DEBUG: Task 30 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/e971ce1a-a910-421d-940a-f8e2b07912fe_isrForCrosstalkSources_1124087_11.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/e971ce1a-a910-421d-940a-f8e2b07912fe_isrForCrosstalkSources_1124087_11.stderr parsl.dataflow.dflow INFO: Task 19 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ebf6315a054042cf1c1838147df76435 with result from task 19 parsl.dataflow.dflow DEBUG: Task 19 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/7e12f0e2-fad8-47e5-a5ac-654bd55622ce_isrForCrosstalkSources_1124087_15.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/7e12f0e2-fad8-47e5-a5ac-654bd55622ce_isrForCrosstalkSources_1124087_15.stderr parsl.dataflow.dflow INFO: Task 10 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 831abaf6e2367b82e0d006fb984c10b3 with result from task 10 parsl.dataflow.dflow DEBUG: Task 10 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/a2e2db14-4b39-4851-80b2-119fd4283b5d_isrForCrosstalkSources_1123956_42.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/a2e2db14-4b39-4851-80b2-119fd4283b5d_isrForCrosstalkSources_1123956_42.stderr parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a11b07f5fc390e58ed267053ddc33c84 with result from task 27 parsl.dataflow.dflow DEBUG: Task 27 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/5c10d107-008b-4812-93f0-76e93b2f5d73_isrForCrosstalkSources_1123954_22.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/5c10d107-008b-4812-93f0-76e93b2f5d73_isrForCrosstalkSources_1123954_22.stderr parsl.dataflow.dflow INFO: Task 28 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 19b139e8ecc6fba0d8d07f42e3999359 with result from task 28 parsl.dataflow.dflow DEBUG: Task 28 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/d3c62696-6396-4c76-814d-11785eb5c23c_isrForCrosstalkSources_1123954_16.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/d3c62696-6396-4c76-814d-11785eb5c23c_isrForCrosstalkSources_1123954_16.stderr parsl.dataflow.dflow INFO: Task 26 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14eb09a0eb5cbfa0a57691227fcced21 with result from task 26 parsl.dataflow.dflow DEBUG: Task 26 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/575c2233-1f47-4acf-895e-3bad555ec38d_isrForCrosstalkSources_1123954_23.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/575c2233-1f47-4acf-895e-3bad555ec38d_isrForCrosstalkSources_1123954_23.stderr parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24d78e4ed3286ee133b13868b72125ee with result from task 13 parsl.dataflow.dflow DEBUG: Task 13 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/0940180a-c6f7-49da-b41e-ef73a05b853d_isrForCrosstalkSources_1123958_10.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/0940180a-c6f7-49da-b41e-ef73a05b853d_isrForCrosstalkSources_1123958_10.stderr parsl.dataflow.dflow INFO: Task 5 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0002a9b7f4fa158ecfd8f315ca96cac with result from task 5 parsl.dataflow.dflow DEBUG: Task 5 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/9c743906-f335-4e5e-b6bf-a72c8ab6e2f0_isrForCrosstalkSources_1123958_57.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/9c743906-f335-4e5e-b6bf-a72c8ab6e2f0_isrForCrosstalkSources_1123958_57.stderr parsl.dataflow.dflow INFO: Task 23 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1f3edf8048b95206b281c1e3eec6754 with result from task 23 parsl.dataflow.dflow DEBUG: Task 23 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/c72b5be8-277d-4a53-8f8e-73388aa72d50_isrForCrosstalkSources_1123956_40.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/c72b5be8-277d-4a53-8f8e-73388aa72d50_isrForCrosstalkSources_1123956_40.stderr parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae56e4d2252b9d60930b274abc8063cc with result from task 14 parsl.dataflow.dflow DEBUG: Task 14 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/5d48e1e2-a9cb-47ec-9223-766e800fd099_isrForCrosstalkSources_1123954_33.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/5d48e1e2-a9cb-47ec-9223-766e800fd099_isrForCrosstalkSources_1123954_33.stderr parsl.dataflow.dflow INFO: Task 25 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbfa4535c55d92836a816b751cad62c0 with result from task 25 parsl.dataflow.dflow DEBUG: Task 25 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/ccf3160c-e231-4dd8-a9b7-a6fbca05db27_isrForCrosstalkSources_1123954_50.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/ccf3160c-e231-4dd8-a9b7-a6fbca05db27_isrForCrosstalkSources_1123954_50.stderr parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93c06abf8b1dfd76e72498f38b420309 with result from task 21 parsl.dataflow.dflow DEBUG: Task 21 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/276d2e56-ea67-4dd7-82c5-10fec5f48356_isrForCrosstalkSources_1123954_32.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/276d2e56-ea67-4dd7-82c5-10fec5f48356_isrForCrosstalkSources_1123954_32.stderr parsl.dataflow.dflow INFO: Task 31 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4654d4715aea011370094a6a54e8dc8 with result from task 31 parsl.dataflow.dflow DEBUG: Task 31 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/80efa6f7-3978-4edd-a9a1-51b099d78d1d_isrForCrosstalkSources_1123958_55.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/80efa6f7-3978-4edd-a9a1-51b099d78d1d_isrForCrosstalkSources_1123958_55.stderr parsl.dataflow.dflow INFO: Task 32 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22628a72f6990044d74b7125057591e5 with result from task 32 parsl.dataflow.dflow DEBUG: Task 32 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/bd4ec73e-2b0e-4423-bbbf-039032933cdd_isrForCrosstalkSources_1123958_34.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/bd4ec73e-2b0e-4423-bbbf-039032933cdd_isrForCrosstalkSources_1123958_34.stderr parsl.dataflow.dflow INFO: Task 22 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 020020d7740f216ec5823a1af4e5705b with result from task 22 parsl.dataflow.dflow DEBUG: Task 22 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/84afb1e2-b8cf-4525-aee9-1df02e39f4ca_isrForCrosstalkSources_1123958_28.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/84afb1e2-b8cf-4525-aee9-1df02e39f4ca_isrForCrosstalkSources_1123958_28.stderr parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ebdc7d571983ea4af38a15d519059fb with result from task 33 parsl.dataflow.dflow DEBUG: Task 33 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/ef2ab5a8-4ad5-46e3-ad47-e8bef77ed6dc_isrForCrosstalkSources_1123954_6.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/ef2ab5a8-4ad5-46e3-ad47-e8bef77ed6dc_isrForCrosstalkSources_1123954_6.stderr parsl.dataflow.dflow INFO: Task 16 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90525e4ce4fa3788935b5a92857956fb with result from task 16 parsl.dataflow.dflow DEBUG: Task 16 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/e337d022-3bc3-4a07-b864-bc966c1be1a1_isrForCrosstalkSources_1124087_51.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/e337d022-3bc3-4a07-b864-bc966c1be1a1_isrForCrosstalkSources_1124087_51.stderr parsl.dataflow.dflow INFO: Task 11 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21138e331816e7f585ca978d73fd4a53 with result from task 11 parsl.dataflow.dflow DEBUG: Task 11 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/6f46e39e-685a-4b1a-a5ef-7a10394dd121_isrForCrosstalkSources_1124087_56.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/6f46e39e-685a-4b1a-a5ef-7a10394dd121_isrForCrosstalkSources_1124087_56.stderr parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d59264b38bbdea5d6a59e5385a0bbb7 with result from task 24 parsl.dataflow.dflow DEBUG: Task 24 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/6fe2aa92-7d72-40a9-9665-45c65f38b8b7_isrForCrosstalkSources_1124087_12.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/6fe2aa92-7d72-40a9-9665-45c65f38b8b7_isrForCrosstalkSources_1124087_12.stderr parsl.dataflow.dflow INFO: Task 4 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 133334d3a88bbcfcfd73d3d3cd288750 with result from task 4 parsl.dataflow.dflow DEBUG: Task 4 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/b4252b57-8f2a-4844-94c9-4cb2add53bb7_isrForCrosstalkSources_1124087_8.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/b4252b57-8f2a-4844-94c9-4cb2add53bb7_isrForCrosstalkSources_1124087_8.stderr parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f5af073de4dd956c129c19c723dedf0 with result from task 18 parsl.dataflow.dflow DEBUG: Task 18 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/6ad90c32-3471-459f-867a-4502a1c9db02_isrForCrosstalkSources_1124087_5.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/6ad90c32-3471-459f-867a-4502a1c9db02_isrForCrosstalkSources_1124087_5.stderr parsl.dataflow.dflow INFO: Task 17 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f98e39073b9530098834a3f44c3cca6 with result from task 17 parsl.dataflow.dflow DEBUG: Task 17 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/02eccca7-c5aa-4cff-a064-5b3124ebae16_isrForCrosstalkSources_1124087_17.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/02eccca7-c5aa-4cff-a064-5b3124ebae16_isrForCrosstalkSources_1124087_17.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 206 parsl.dataflow.strategy DEBUG: Executor multi has 206 active tasks, 61/1 running/pending blocks, and 97 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22988724935824 parsl.dataflow.dflow INFO: Task 75 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 327b536576bbac117c7fb5a9b3367afb with result from task 75 parsl.dataflow.dflow DEBUG: Task 75 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/70a8a9b0-e325-4de5-b36a-fa4287286950_isrForCrosstalkSources_1123958_12.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/70a8a9b0-e325-4de5-b36a-fa4287286950_isrForCrosstalkSources_1123958_12.stderr parsl.dataflow.dflow INFO: Task 76 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa7b93a5a0fa4f531b2c9fd6616cc231 with result from task 76 parsl.dataflow.dflow DEBUG: Task 76 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/9dfb1ae0-4c55-4c36-a00b-fceb83a72ebe_isrForCrosstalkSources_1123954_41.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/9dfb1ae0-4c55-4c36-a00b-fceb83a72ebe_isrForCrosstalkSources_1123954_41.stderr 2024-03-07 03:35:02 proc_lsst.multi:146 [INFO] found job 766 in provider local proc_lsst.multi INFO: found job 766 in provider local 2024-03-07 03:35:02 proc_lsst.multi:146 [INFO] found job 16884172 in provider astro proc_lsst.multi INFO: found job 16884172 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:03 proc_lsst.multi:146 [INFO] found job 16884173 in provider astro proc_lsst.multi INFO: found job 16884173 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:03 proc_lsst.multi:146 [INFO] found job 16884174 in provider astro proc_lsst.multi INFO: found job 16884174 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:04 proc_lsst.multi:146 [INFO] found job 16884175 in provider astro proc_lsst.multi INFO: found job 16884175 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:04 proc_lsst.multi:146 [INFO] found job 16884176 in provider astro proc_lsst.multi INFO: found job 16884176 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:04 proc_lsst.multi:146 [INFO] found job 16884177 in provider astro proc_lsst.multi INFO: found job 16884177 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:04 proc_lsst.multi:146 [INFO] found job 16884178 in provider astro proc_lsst.multi INFO: found job 16884178 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:05 proc_lsst.multi:146 [INFO] found job 16884179 in provider astro proc_lsst.multi INFO: found job 16884179 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:05 proc_lsst.multi:146 [INFO] found job 16884180 in provider astro proc_lsst.multi INFO: found job 16884180 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:05 proc_lsst.multi:146 [INFO] found job 16884181 in provider astro proc_lsst.multi INFO: found job 16884181 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.dataflow.dflow INFO: Task 83 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 830965738b067cef64dcb2844f615cd9 with result from task 83 parsl.dataflow.dflow DEBUG: Task 83 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/bbbc92ed-cfa8-47a8-887e-ee2fde097336_isrForCrosstalkSources_1123954_51.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/bbbc92ed-cfa8-47a8-887e-ee2fde097336_isrForCrosstalkSources_1123954_51.stderr parsl.dataflow.dflow INFO: Task 86 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59410c807a22c55a476e935caa36e33a with result from task 86 parsl.dataflow.dflow DEBUG: Task 86 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/6bf6a007-adfb-4f29-b546-33b51ef1ef44_isrForCrosstalkSources_1123954_5.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/6bf6a007-adfb-4f29-b546-33b51ef1ef44_isrForCrosstalkSources_1123954_5.stderr parsl.dataflow.dflow INFO: Task 84 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0f3e7a421edd789af94364bcc575599 with result from task 84 parsl.dataflow.dflow DEBUG: Task 84 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/3cb27e9f-30a8-472a-a971-55594cb6d546_isrForCrosstalkSources_1123954_12.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/3cb27e9f-30a8-472a-a971-55594cb6d546_isrForCrosstalkSources_1123954_12.stderr parsl.dataflow.dflow INFO: Task 79 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e59e7be6885c2d518184fd56bd032a2 with result from task 79 parsl.dataflow.dflow DEBUG: Task 79 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/fac16bcc-dd5e-442a-b61f-5ca8d78cfd09_isrForCrosstalkSources_1123956_23.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/fac16bcc-dd5e-442a-b61f-5ca8d78cfd09_isrForCrosstalkSources_1123956_23.stderr parsl.dataflow.dflow INFO: Task 82 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f723c95841942f8a1341beab91c533e9 with result from task 82 parsl.dataflow.dflow DEBUG: Task 82 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/fcba0bde-fc56-463f-a71a-af7540ee366c_isrForCrosstalkSources_1123958_22.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/fcba0bde-fc56-463f-a71a-af7540ee366c_isrForCrosstalkSources_1123958_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:05 proc_lsst.multi:146 [INFO] found job 16884182 in provider astro proc_lsst.multi INFO: found job 16884182 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.dataflow.dflow INFO: Task 81 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6842875bacc61b41064ef481781d1464 with result from task 81 parsl.dataflow.dflow DEBUG: Task 81 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/9dd5ccc0-fa0a-43fa-8376-ac56c34feaab_isrForCrosstalkSources_1123958_9.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/9dd5ccc0-fa0a-43fa-8376-ac56c34feaab_isrForCrosstalkSources_1123958_9.stderr parsl.dataflow.dflow INFO: Task 85 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc504ef86d000e012cadc9889b8dee17 with result from task 85 parsl.dataflow.dflow DEBUG: Task 85 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/3f60ea24-28a8-40c5-ac2e-3529594d7e6c_isrForCrosstalkSources_1123954_62.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/3f60ea24-28a8-40c5-ac2e-3529594d7e6c_isrForCrosstalkSources_1123954_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:06 proc_lsst.multi:146 [INFO] found job 16884183 in provider astro proc_lsst.multi INFO: found job 16884183 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.dataflow.dflow INFO: Task 90 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd7cd990a8f49d5a1e9dade52be736e5 with result from task 90 parsl.dataflow.dflow DEBUG: Task 90 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/4549f735-cb71-41c9-8259-cb5bc3355852_isrForCrosstalkSources_1123954_19.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/4549f735-cb71-41c9-8259-cb5bc3355852_isrForCrosstalkSources_1123954_19.stderr parsl.dataflow.dflow INFO: Task 91 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a91210bfe8357ef3167b75a156a0784 with result from task 91 parsl.dataflow.dflow DEBUG: Task 91 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/2988226c-ec86-4762-acc6-acab7176638d_isrForCrosstalkSources_1123954_10.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/2988226c-ec86-4762-acc6-acab7176638d_isrForCrosstalkSources_1123954_10.stderr parsl.dataflow.dflow INFO: Task 106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39c955f9482c5dbcdab9489a83159520 with result from task 106 parsl.dataflow.dflow DEBUG: Task 106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/fb496018-2590-4a36-b5ba-34beb55528d5_isrForCrosstalkSources_1123954_46.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/fb496018-2590-4a36-b5ba-34beb55528d5_isrForCrosstalkSources_1123954_46.stderr parsl.dataflow.dflow INFO: Task 102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4bc824b6de1c2200872cb4715f5a9de1 with result from task 102 parsl.dataflow.dflow DEBUG: Task 102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/3aae3930-48c0-4c52-952b-c4cf48684818_isrForCrosstalkSources_1123954_9.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/3aae3930-48c0-4c52-952b-c4cf48684818_isrForCrosstalkSources_1123954_9.stderr parsl.dataflow.dflow INFO: Task 101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72d6f68fe5491e5856181da6772a7be7 with result from task 101 parsl.dataflow.dflow DEBUG: Task 101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/d923848f-e9b4-4669-8a86-d68491eff739_isrForCrosstalkSources_1123958_5.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/d923848f-e9b4-4669-8a86-d68491eff739_isrForCrosstalkSources_1123958_5.stderr parsl.dataflow.dflow INFO: Task 88 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa1302db87ec6f18942e934d7d70366e with result from task 88 parsl.dataflow.dflow DEBUG: Task 88 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/80ba5282-8dfc-4970-9904-3a9c49229f09_isrForCrosstalkSources_1123958_11.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/80ba5282-8dfc-4970-9904-3a9c49229f09_isrForCrosstalkSources_1123958_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:06 proc_lsst.multi:146 [INFO] found job 16884184 in provider astro proc_lsst.multi INFO: found job 16884184 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.dataflow.dflow INFO: Task 95 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63f97caf01e8cca628f2b30b6ac18d68 with result from task 95 parsl.dataflow.dflow DEBUG: Task 95 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/7082a77c-9ef4-423d-95a2-2b96853150ad_isrForCrosstalkSources_1123954_37.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/7082a77c-9ef4-423d-95a2-2b96853150ad_isrForCrosstalkSources_1123954_37.stderr parsl.dataflow.dflow INFO: Task 87 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1af97dc7391513a04cae702888d87a63 with result from task 87 parsl.dataflow.dflow DEBUG: Task 87 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/453aac3f-0299-4253-af66-92ac6b3d3bff_isrForCrosstalkSources_1123958_7.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/453aac3f-0299-4253-af66-92ac6b3d3bff_isrForCrosstalkSources_1123958_7.stderr parsl.dataflow.dflow INFO: Task 92 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fafa903d4946adf4801e92a3641ce34c with result from task 92 parsl.dataflow.dflow DEBUG: Task 92 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/d660f5a9-e85f-4353-bbb1-ebe62dfe9dd9_isrForCrosstalkSources_1123956_20.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/d660f5a9-e85f-4353-bbb1-ebe62dfe9dd9_isrForCrosstalkSources_1123956_20.stderr parsl.dataflow.dflow INFO: Task 98 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc500fa263102f0f952d6397368bdd9e with result from task 98 parsl.dataflow.dflow DEBUG: Task 98 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/a0630e0a-e076-493a-8729-890d1fa73cbc_isrForCrosstalkSources_1123954_38.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/a0630e0a-e076-493a-8729-890d1fa73cbc_isrForCrosstalkSources_1123954_38.stderr parsl.dataflow.dflow INFO: Task 99 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65b43fb1c2908f0ee0d7fd82545417d7 with result from task 99 parsl.dataflow.dflow DEBUG: Task 99 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/1ac3a382-b337-450a-be30-afa68a0ff057_isrForCrosstalkSources_1123954_42.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/1ac3a382-b337-450a-be30-afa68a0ff057_isrForCrosstalkSources_1123954_42.stderr parsl.dataflow.dflow INFO: Task 104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a5ba1773ec2d52827d9f83f8c3dbf24 with result from task 104 parsl.dataflow.dflow DEBUG: Task 104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/a39b13f8-51f5-465c-be1d-abde392b9fb6_isrForCrosstalkSources_1123954_55.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/a39b13f8-51f5-465c-be1d-abde392b9fb6_isrForCrosstalkSources_1123954_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:06 proc_lsst.multi:146 [INFO] found job 16884185 in provider astro proc_lsst.multi INFO: found job 16884185 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.dataflow.dflow INFO: Task 105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b7a9681aa47321d65b664c4d53f8eb8 with result from task 105 parsl.dataflow.dflow DEBUG: Task 105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/2d4dec71-b517-4130-a0d0-638f24458f36_isrForCrosstalkSources_1123958_53.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/2d4dec71-b517-4130-a0d0-638f24458f36_isrForCrosstalkSources_1123958_53.stderr parsl.dataflow.dflow INFO: Task 80 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c322ccb014805c24eb05724adffb3872 with result from task 80 parsl.dataflow.dflow DEBUG: Task 80 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/f8796ce1-a33f-450e-8274-17c21ba4c9ef_isrForCrosstalkSources_1124087_4.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/f8796ce1-a33f-450e-8274-17c21ba4c9ef_isrForCrosstalkSources_1124087_4.stderr parsl.dataflow.dflow INFO: Task 94 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f2c4b8c0e90bfacc0b9ffe36c76751d with result from task 94 parsl.dataflow.dflow DEBUG: Task 94 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/08b752c2-c33f-4b4b-b4e0-e83ae7f2164d_isrForCrosstalkSources_1123954_20.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/08b752c2-c33f-4b4b-b4e0-e83ae7f2164d_isrForCrosstalkSources_1123954_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:07 proc_lsst.multi:146 [INFO] found job 16884186 in provider astro proc_lsst.multi INFO: found job 16884186 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.dataflow.dflow INFO: Task 77 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad0ddf5872a46314d93055f4e171ab19 with result from task 77 parsl.dataflow.dflow DEBUG: Task 77 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/e87e6255-4323-48ce-abfa-ec6480abf5ee_isrForCrosstalkSources_1124087_47.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/e87e6255-4323-48ce-abfa-ec6480abf5ee_isrForCrosstalkSources_1124087_47.stderr parsl.dataflow.dflow INFO: Task 78 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da3e8badf98c890544389a279838a98e with result from task 78 parsl.dataflow.dflow DEBUG: Task 78 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/a0d52380-1bfd-4e26-97d1-dff0ec256b42_isrForCrosstalkSources_1124087_43.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/a0d52380-1bfd-4e26-97d1-dff0ec256b42_isrForCrosstalkSources_1124087_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:07 proc_lsst.multi:146 [INFO] found job 16884187 in provider astro proc_lsst.multi INFO: found job 16884187 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.dataflow.dflow INFO: Task 89 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3263f19f18bfec32dffd44a8fb7f58e0 with result from task 89 parsl.dataflow.dflow DEBUG: Task 89 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/3c56078a-bb66-4d56-bdae-a48975735bea_isrForCrosstalkSources_1124087_16.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/3c56078a-bb66-4d56-bdae-a48975735bea_isrForCrosstalkSources_1124087_16.stderr parsl.dataflow.dflow INFO: Task 97 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b64abcdfeefd28a8d784a869948068b with result from task 97 parsl.dataflow.dflow DEBUG: Task 97 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/59cc7fd9-0794-4683-8f4d-1c7d31af6a83_isrForCrosstalkSources_1124087_46.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/59cc7fd9-0794-4683-8f4d-1c7d31af6a83_isrForCrosstalkSources_1124087_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:07 proc_lsst.multi:146 [INFO] found job 16884188 in provider astro proc_lsst.multi INFO: found job 16884188 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.dataflow.dflow INFO: Task 100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49d8ba424cd6467b0cc1c9e2516b258f with result from task 100 parsl.dataflow.dflow DEBUG: Task 100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/676d10d8-ef04-4ce0-b7a9-234df7eb8592_isrForCrosstalkSources_1124087_30.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/676d10d8-ef04-4ce0-b7a9-234df7eb8592_isrForCrosstalkSources_1124087_30.stderr parsl.dataflow.dflow INFO: Task 103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 88491b14893bbceeff929a9158c05af4 with result from task 103 parsl.dataflow.dflow DEBUG: Task 103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/69f3c2b9-9b80-405c-bd26-eedd87ad7bc0_isrForCrosstalkSources_1124087_40.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/69f3c2b9-9b80-405c-bd26-eedd87ad7bc0_isrForCrosstalkSources_1124087_40.stderr parsl.dataflow.dflow INFO: Task 93 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48d991c19eec7f0bea16af7623b90330 with result from task 93 parsl.dataflow.dflow DEBUG: Task 93 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/98fa5e2a-f825-4c7c-9e28-b8782e9670a6_isrForCrosstalkSources_1124087_7.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/98fa5e2a-f825-4c7c-9e28-b8782e9670a6_isrForCrosstalkSources_1124087_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:08 proc_lsst.multi:146 [INFO] found job 16884189 in provider astro proc_lsst.multi INFO: found job 16884189 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:08 proc_lsst.multi:146 [INFO] found job 16884190 in provider astro proc_lsst.multi INFO: found job 16884190 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.dataflow.dflow INFO: Task 96 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d33ea46c0d61979cd4e12fea2a155dc with result from task 96 parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:08 proc_lsst.multi:146 [INFO] found job 16884191 in provider astro proc_lsst.multi INFO: found job 16884191 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.dataflow.dflow DEBUG: Task 96 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/ac707209-f50f-4d9e-b9b9-5b9d12621d95_isrForCrosstalkSources_1124087_14.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/ac707209-f50f-4d9e-b9b9-5b9d12621d95_isrForCrosstalkSources_1124087_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:09 proc_lsst.multi:146 [INFO] found job 16884192 in provider astro proc_lsst.multi INFO: found job 16884192 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:09 proc_lsst.multi:146 [INFO] found job 16884193 in provider astro proc_lsst.multi INFO: found job 16884193 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:09 proc_lsst.multi:146 [INFO] found job 16884194 in provider astro proc_lsst.multi INFO: found job 16884194 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:10 proc_lsst.multi:146 [INFO] found job 16884195 in provider astro proc_lsst.multi INFO: found job 16884195 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.dataflow.dflow INFO: Task 139 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4954305c43be0d881acdcc9aed95fdba with result from task 139 parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:10 proc_lsst.multi:146 [INFO] found job 16884196 in provider astro proc_lsst.multi INFO: found job 16884196 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.dataflow.dflow DEBUG: Task 139 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/8b68e1fd-6156-4c09-b26d-4f6de53c72ae_isrForCrosstalkSources_1124087_58.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/8b68e1fd-6156-4c09-b26d-4f6de53c72ae_isrForCrosstalkSources_1124087_58.stderr parsl.dataflow.dflow INFO: Task 140 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ecc9caed903b9fcd07a1f810c8ebc7d with result from task 140 parsl.dataflow.dflow DEBUG: Task 140 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/08096885-3c9e-4b84-b8dc-f76de27f94f7_isrForCrosstalkSources_1124087_10.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/08096885-3c9e-4b84-b8dc-f76de27f94f7_isrForCrosstalkSources_1124087_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:12 proc_lsst.multi:146 [INFO] found job 16884197 in provider astro proc_lsst.multi INFO: found job 16884197 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:12 proc_lsst.multi:146 [INFO] found job 16884203 in provider astro proc_lsst.multi INFO: found job 16884203 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:13 proc_lsst.multi:146 [INFO] found job 16884204 in provider astro proc_lsst.multi INFO: found job 16884204 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:13 proc_lsst.multi:146 [INFO] found job 16884205 in provider astro proc_lsst.multi INFO: found job 16884205 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:13 proc_lsst.multi:146 [INFO] found job 16884206 in provider astro proc_lsst.multi INFO: found job 16884206 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:14 proc_lsst.multi:146 [INFO] found job 16884207 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884207 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884207,16884208,16884209,16884210,16884211' parsl.providers.slurm.slurm DEBUG: squeue returned 16884207 R 16884208 R 16884209 R 16884210 R 16884211 R parsl.providers.slurm.slurm DEBUG: Updating job 16884207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884211 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:14 proc_lsst.multi:146 [INFO] found job 16884208 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884208 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884207,16884208,16884209,16884210,16884211' parsl.providers.slurm.slurm DEBUG: squeue returned 16884207 R 16884208 R 16884209 R 16884210 R 16884211 R parsl.providers.slurm.slurm DEBUG: Updating job 16884207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884211 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:14 proc_lsst.multi:146 [INFO] found job 16884209 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884209 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884207,16884208,16884209,16884210,16884211' parsl.providers.slurm.slurm DEBUG: squeue returned 16884207 R 16884208 R 16884209 R 16884210 R 16884211 R parsl.providers.slurm.slurm DEBUG: Updating job 16884207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884211 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:14 proc_lsst.multi:146 [INFO] found job 16884210 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884210 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884207,16884208,16884209,16884210,16884211' parsl.providers.slurm.slurm DEBUG: squeue returned 16884207 R 16884208 R 16884209 R 16884210 R 16884211 R parsl.providers.slurm.slurm DEBUG: Updating job 16884207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884211 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:15 proc_lsst.multi:146 [INFO] found job 16884211 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884211 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884207,16884208,16884209,16884210,16884211' parsl.providers.slurm.slurm DEBUG: squeue returned 16884207 R 16884208 R 16884209 R 16884210 R 16884211 R parsl.providers.slurm.slurm DEBUG: Updating job 16884207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884211 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:15 proc_lsst.multi:146 [INFO] found job 16884212 in provider escience_a40 proc_lsst.multi INFO: found job 16884212 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:15 proc_lsst.multi:146 [INFO] found job 16884213 in provider escience_a40 proc_lsst.multi INFO: found job 16884213 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:16 proc_lsst.multi:146 [INFO] found job 16884214 in provider escience_a40 proc_lsst.multi INFO: found job 16884214 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.dataflow.dflow INFO: Task 146 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66a780cdb3792a3c1a4d38354723cc9e with result from task 146 parsl.dataflow.dflow DEBUG: Task 146 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/3a84d0f3-35f1-4670-9b0e-3b730323a191_isrForCrosstalkSources_1123956_32.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/3a84d0f3-35f1-4670-9b0e-3b730323a191_isrForCrosstalkSources_1123956_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:16 proc_lsst.multi:146 [INFO] found job 16884215 in provider escience_a40 proc_lsst.multi INFO: found job 16884215 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:16 proc_lsst.multi:146 [INFO] found job 16884216 in provider escience_a40 proc_lsst.multi INFO: found job 16884216 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.dataflow.dflow INFO: Task 164 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cef4ae02be2240d2a33cc23a477b7237 with result from task 164 parsl.dataflow.dflow DEBUG: Task 164 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/4c6fe503-8b8d-479a-8b2c-3d2831114e74_isrForCrosstalkSources_1123954_45.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/4c6fe503-8b8d-479a-8b2c-3d2831114e74_isrForCrosstalkSources_1123954_45.stderr parsl.dataflow.dflow INFO: Task 145 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb2915c62e7f6ddd55d433107bb9c1ab with result from task 145 parsl.dataflow.dflow DEBUG: Task 145 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/b70028bf-0624-4190-af8b-0d05cb767470_isrForCrosstalkSources_1123956_1.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/b70028bf-0624-4190-af8b-0d05cb767470_isrForCrosstalkSources_1123956_1.stderr parsl.dataflow.dflow INFO: Task 147 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e243a10f79b9648cf2d41b20f303d38 with result from task 147 parsl.dataflow.dflow DEBUG: Task 147 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/820a9ddb-86bc-4668-910d-1cc0fc4151c8_isrForCrosstalkSources_1123958_36.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/820a9ddb-86bc-4668-910d-1cc0fc4151c8_isrForCrosstalkSources_1123958_36.stderr parsl.dataflow.dflow INFO: Task 141 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ca4bb816af0bed93909ab08ca1e9b05 with result from task 141 parsl.dataflow.dflow DEBUG: Task 141 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/959ed997-37d3-4cd8-91b1-d644a344d235_isrForCrosstalkSources_1123956_35.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/959ed997-37d3-4cd8-91b1-d644a344d235_isrForCrosstalkSources_1123956_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:17 proc_lsst.multi:146 [INFO] found job 16884217 in provider escience_a40 proc_lsst.multi INFO: found job 16884217 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.dataflow.dflow INFO: Task 142 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9eca8600115e7ff7a0a1850b90bb2fba with result from task 142 parsl.dataflow.dflow DEBUG: Task 142 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/e2e29434-1d4a-4bd8-9ada-0a59a0497c21_isrForCrosstalkSources_1123958_61.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/e2e29434-1d4a-4bd8-9ada-0a59a0497c21_isrForCrosstalkSources_1123958_61.stderr parsl.dataflow.dflow INFO: Task 152 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c03cb10d37eaae49567126c07df92135 with result from task 152 parsl.dataflow.dflow DEBUG: Task 152 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/e7b60cdf-c83f-4d6a-806f-6a04e90c8faf_isrForCrosstalkSources_1123958_39.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/e7b60cdf-c83f-4d6a-806f-6a04e90c8faf_isrForCrosstalkSources_1123958_39.stderr parsl.dataflow.dflow INFO: Task 154 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df4e39c4900c7149a3f4831bdbc40d73 with result from task 154 parsl.dataflow.dflow DEBUG: Task 154 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/a7844584-a529-4e83-89d0-33e4154104fd_isrForCrosstalkSources_1123954_13.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/a7844584-a529-4e83-89d0-33e4154104fd_isrForCrosstalkSources_1123954_13.stderr parsl.dataflow.dflow INFO: Task 156 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5319a30827caab7d6e81a170647ebe6f with result from task 156 parsl.dataflow.dflow DEBUG: Task 156 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/6ddc7722-37df-443f-870a-b6c599c3e933_isrForCrosstalkSources_1123956_18.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/6ddc7722-37df-443f-870a-b6c599c3e933_isrForCrosstalkSources_1123956_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:17 proc_lsst.multi:146 [INFO] found job 16884218 in provider escience_a40 proc_lsst.multi INFO: found job 16884218 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.dataflow.dflow INFO: Task 151 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f558018fb0c4fc8d605ac89c1ea4894 with result from task 151 parsl.dataflow.dflow DEBUG: Task 151 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/43db8f5e-25e8-4402-938f-7a4e38a82915_isrForCrosstalkSources_1123954_18.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/43db8f5e-25e8-4402-938f-7a4e38a82915_isrForCrosstalkSources_1123954_18.stderr parsl.dataflow.dflow INFO: Task 162 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad0bcaacc7924825f5642efaf1a807b6 with result from task 162 parsl.dataflow.dflow DEBUG: Task 162 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/7050a1e5-189a-49d7-9515-a87322f13481_isrForCrosstalkSources_1123958_13.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/7050a1e5-189a-49d7-9515-a87322f13481_isrForCrosstalkSources_1123958_13.stderr parsl.dataflow.dflow INFO: Task 153 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b4109bc98344456eec0f62bd16b3059 with result from task 153 parsl.dataflow.dflow DEBUG: Task 153 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/485bcfcf-358c-4d2d-aaa6-5dd3d7c56fc1_isrForCrosstalkSources_1123958_43.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/485bcfcf-358c-4d2d-aaa6-5dd3d7c56fc1_isrForCrosstalkSources_1123958_43.stderr parsl.dataflow.dflow INFO: Task 158 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2413d28d03c329eb7fbe908e6c7897bf with result from task 158 parsl.dataflow.dflow DEBUG: Task 158 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/5cb424fa-207a-4a51-b535-f3e26db5d959_isrForCrosstalkSources_1123954_1.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/5cb424fa-207a-4a51-b535-f3e26db5d959_isrForCrosstalkSources_1123954_1.stderr parsl.dataflow.dflow INFO: Task 159 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57d70f4bcee331587be173989ebc2fe5 with result from task 159 parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:17 proc_lsst.multi:146 [INFO] found job 16884219 in provider escience_a40 proc_lsst.multi INFO: found job 16884219 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.dataflow.dflow DEBUG: Task 159 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/b91ab98a-d51e-4df7-8c98-bf2d2f62216b_isrForCrosstalkSources_1123956_31.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/b91ab98a-d51e-4df7-8c98-bf2d2f62216b_isrForCrosstalkSources_1123956_31.stderr parsl.dataflow.dflow INFO: Task 155 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aedf537eba954f741fc5fc00264d3088 with result from task 155 parsl.dataflow.dflow DEBUG: Task 155 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/0c6d59c0-52ea-45aa-ae86-c593f652edf1_isrForCrosstalkSources_1123954_52.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/0c6d59c0-52ea-45aa-ae86-c593f652edf1_isrForCrosstalkSources_1123954_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:18 proc_lsst.multi:146 [INFO] found job 16884220 in provider escience_a40 proc_lsst.multi INFO: found job 16884220 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.dataflow.dflow INFO: Task 165 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cefa85e85a4cd7af84c40529976c2a27 with result from task 165 parsl.dataflow.dflow DEBUG: Task 165 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/3397608d-12da-499e-868d-6bc78715865a_isrForCrosstalkSources_1123954_4.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/3397608d-12da-499e-868d-6bc78715865a_isrForCrosstalkSources_1123954_4.stderr parsl.dataflow.dflow INFO: Task 149 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8359d1c3d9ea42a0b9ef8c1e93a4e68 with result from task 149 parsl.dataflow.dflow DEBUG: Task 149 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/9bb6f0e8-5d70-4870-8ef7-45af0f1b8b8a_isrForCrosstalkSources_1123956_4.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/9bb6f0e8-5d70-4870-8ef7-45af0f1b8b8a_isrForCrosstalkSources_1123956_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:18 proc_lsst.multi:146 [INFO] found job 16884221 in provider escience_a40 proc_lsst.multi INFO: found job 16884221 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.dataflow.dflow INFO: Task 163 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7ee8299de06901c4ba941022dd72ba2 with result from task 163 parsl.dataflow.dflow DEBUG: Task 163 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/076e1b5e-0cf3-41a4-aaf7-53fd69b67e33_isrForCrosstalkSources_1124087_19.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/076e1b5e-0cf3-41a4-aaf7-53fd69b67e33_isrForCrosstalkSources_1124087_19.stderr parsl.dataflow.dflow INFO: Task 144 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd74121f62acffae7c967622b4aef5fa with result from task 144 parsl.dataflow.dflow DEBUG: Task 144 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/9f46e9e5-5a83-4dcb-853d-bed80078eaa1_isrForCrosstalkSources_1124087_45.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/9f46e9e5-5a83-4dcb-853d-bed80078eaa1_isrForCrosstalkSources_1124087_45.stderr parsl.dataflow.dflow INFO: Task 169 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d371c9bf167788268ce4fbefc9bbfcd with result from task 169 parsl.dataflow.dflow DEBUG: Task 169 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/61cd4253-fe6c-4bff-92a8-1a628353454a_isrForCrosstalkSources_1123958_33.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/61cd4253-fe6c-4bff-92a8-1a628353454a_isrForCrosstalkSources_1123958_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:18 proc_lsst.multi:146 [INFO] found job 16884222 in provider escience_a40 proc_lsst.multi INFO: found job 16884222 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.dataflow.dflow INFO: Task 171 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ac3c4dc4cbe71b9564f9b25f537d484 with result from task 171 parsl.dataflow.dflow DEBUG: Task 171 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/e9f4dc88-b052-456a-93db-28da973ae5cf_isrForCrosstalkSources_1123958_44.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/e9f4dc88-b052-456a-93db-28da973ae5cf_isrForCrosstalkSources_1123958_44.stderr parsl.dataflow.dflow INFO: Task 168 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f84defbdd9d09332864ca0f75b9381af with result from task 168 parsl.dataflow.dflow DEBUG: Task 168 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/36dd93cc-4e88-49fe-a812-542d9874bb5d_isrForCrosstalkSources_1123954_53.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/36dd93cc-4e88-49fe-a812-542d9874bb5d_isrForCrosstalkSources_1123954_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:18 proc_lsst.multi:146 [INFO] found job 16884223 in provider escience_a40 proc_lsst.multi INFO: found job 16884223 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:19 proc_lsst.multi:146 [INFO] found job 16884224 in provider escience_a40 proc_lsst.multi INFO: found job 16884224 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:19 proc_lsst.multi:146 [INFO] found job 16884225 in provider escience_a40 proc_lsst.multi INFO: found job 16884225 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.dataflow.dflow INFO: Task 166 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 899880df6e1f7967998a016fda9453a1 with result from task 166 parsl.dataflow.dflow DEBUG: Task 166 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/dc8afbc4-8463-4a9f-90a4-6921554b542a_isrForCrosstalkSources_1123956_16.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/dc8afbc4-8463-4a9f-90a4-6921554b542a_isrForCrosstalkSources_1123956_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:19 proc_lsst.multi:146 [INFO] found job 16884226 in provider escience_a40 proc_lsst.multi INFO: found job 16884226 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.dataflow.dflow INFO: Task 160 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33e93a1e2b9e5fa9e733524742396b9a with result from task 160 parsl.dataflow.dflow DEBUG: Task 160 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/63f85edd-16e7-443d-bfc8-5abce2f1c751_isrForCrosstalkSources_1124087_21.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/63f85edd-16e7-443d-bfc8-5abce2f1c751_isrForCrosstalkSources_1124087_21.stderr parsl.dataflow.dflow INFO: Task 170 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 707e8096650c82a624d8a3c1ca07f38c with result from task 170 parsl.dataflow.dflow DEBUG: Task 170 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/e7edfc4c-be51-4249-930e-5c696dfd32d9_isrForCrosstalkSources_1123958_14.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/e7edfc4c-be51-4249-930e-5c696dfd32d9_isrForCrosstalkSources_1123958_14.stderr parsl.dataflow.dflow INFO: Task 172 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6c4ae65efdfa6253bc0e3ecab19b8b5 with result from task 172 parsl.dataflow.dflow DEBUG: Task 172 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/72e88f04-129e-42f2-9a38-522daa6d47b4_isrForCrosstalkSources_1123958_24.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/72e88f04-129e-42f2-9a38-522daa6d47b4_isrForCrosstalkSources_1123958_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:19 proc_lsst.multi:146 [INFO] found job 16884227 in provider escience_a40 proc_lsst.multi INFO: found job 16884227 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:20 proc_lsst.multi:146 [INFO] found job 16884228 in provider escience_a40 proc_lsst.multi INFO: found job 16884228 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:20 proc_lsst.multi:146 [INFO] found job 16884229 in provider escience_a40 proc_lsst.multi INFO: found job 16884229 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.dataflow.dflow INFO: Task 157 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b65f1158e0ee723af51a6012e3093232 with result from task 157 parsl.dataflow.dflow DEBUG: Task 157 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/6a29e527-1df8-44d2-aa32-bcb59d17b89d_isrForCrosstalkSources_1124087_23.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/6a29e527-1df8-44d2-aa32-bcb59d17b89d_isrForCrosstalkSources_1124087_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:20 proc_lsst.multi:146 [INFO] found job 16884230 in provider escience_a40 proc_lsst.multi INFO: found job 16884230 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.dataflow.dflow INFO: Task 167 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79a4f2874d76529b2219035037ced6f2 with result from task 167 parsl.dataflow.dflow DEBUG: Task 167 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/32222add-9750-41e4-8558-d869a6ec042c_isrForCrosstalkSources_1124087_35.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/32222add-9750-41e4-8558-d869a6ec042c_isrForCrosstalkSources_1124087_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:21 proc_lsst.multi:146 [INFO] found job 16884231 in provider escience_a40 proc_lsst.multi INFO: found job 16884231 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:21 proc_lsst.multi:146 [INFO] found job 16884232 in provider checkpoint proc_lsst.multi INFO: found job 16884232 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884232,16884233,16884234,16884235,16884236,16884237' parsl.providers.slurm.slurm DEBUG: squeue returned 16884237 PD 16884232 R 16884233 R 16884234 R 16884235 R 16884236 R parsl.providers.slurm.slurm DEBUG: Updating job 16884237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16884232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884236 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:21 proc_lsst.multi:146 [INFO] found job 16884233 in provider checkpoint proc_lsst.multi INFO: found job 16884233 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884232,16884233,16884234,16884235,16884236,16884237' parsl.providers.slurm.slurm DEBUG: squeue returned 16884237 PD 16884232 R 16884233 R 16884234 R 16884235 R 16884236 R parsl.providers.slurm.slurm DEBUG: Updating job 16884237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16884232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884236 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:22 proc_lsst.multi:146 [INFO] found job 16884234 in provider checkpoint proc_lsst.multi INFO: found job 16884234 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884232,16884233,16884234,16884235,16884236,16884237' parsl.providers.slurm.slurm DEBUG: squeue returned 16884237 PD 16884232 R 16884233 R 16884234 R 16884235 R 16884236 R parsl.providers.slurm.slurm DEBUG: Updating job 16884237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16884232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884236 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:22 proc_lsst.multi:146 [INFO] found job 16884235 in provider checkpoint proc_lsst.multi INFO: found job 16884235 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884232,16884233,16884234,16884235,16884236,16884237' parsl.dataflow.dflow INFO: Task 60 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa0ac73e6a99c0df5156ee8a15dfbd63 with result from task 60 parsl.dataflow.dflow DEBUG: Task 60 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/0bc8f37e-47f8-47ca-ac39-bd68f3d7621a_isrForCrosstalkSources_1123956_13.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/0bc8f37e-47f8-47ca-ac39-bd68f3d7621a_isrForCrosstalkSources_1123956_13.stderr parsl.dataflow.dflow INFO: Task 61 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d6d4cda71002eae7bbf1e5623a1f0fb with result from task 61 parsl.dataflow.dflow DEBUG: Task 61 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/b2defb6e-d680-4b67-86d7-17e91d2cc90b_isrForCrosstalkSources_1123956_45.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/b2defb6e-d680-4b67-86d7-17e91d2cc90b_isrForCrosstalkSources_1123956_45.stderr parsl.dataflow.dflow INFO: Task 59 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac254384f21b3b2055417a87155a722f with result from task 59 parsl.providers.slurm.slurm DEBUG: squeue returned 16884237 PD 16884232 R 16884233 R 16884234 R 16884235 R 16884236 R parsl.providers.slurm.slurm DEBUG: Updating job 16884237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16884232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884236 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:23 proc_lsst.multi:146 [INFO] found job 16884236 in provider checkpoint proc_lsst.multi INFO: found job 16884236 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884232,16884233,16884234,16884235,16884236,16884237' parsl.dataflow.dflow DEBUG: Task 59 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/59b74ac7-08ce-4e6e-8186-a3ef96d91ee7_isrForCrosstalkSources_1123956_6.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/59b74ac7-08ce-4e6e-8186-a3ef96d91ee7_isrForCrosstalkSources_1123956_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884237 PD 16884232 R 16884233 R 16884234 R 16884235 R 16884236 R parsl.providers.slurm.slurm DEBUG: Updating job 16884237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16884232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884236 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:23 proc_lsst.multi:146 [INFO] found job 16884237 in provider checkpoint proc_lsst.multi INFO: found job 16884237 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884232,16884233,16884234,16884235,16884236,16884237' parsl.providers.slurm.slurm DEBUG: squeue returned 16884237 PD 16884232 R 16884233 R 16884234 R 16884235 R 16884236 R parsl.providers.slurm.slurm DEBUG: Updating job 16884237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16884232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884236 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 141 parsl.dataflow.strategy DEBUG: Executor multi has 141 active tasks, 61/1 running/pending blocks, and 97 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22988724935824 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 141 parsl.dataflow.strategy DEBUG: Executor multi has 141 active tasks, 61/1 running/pending blocks, and 97 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22988724935824 parsl.dataflow.dflow INFO: Task 63 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4ef4f5fc433952a20bd37ce82003ad7 with result from task 63 parsl.dataflow.dflow DEBUG: Task 63 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/e103b8ad-111a-4b1c-b1ad-6135abf9f959_isrForCrosstalkSources_1123958_25.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/e103b8ad-111a-4b1c-b1ad-6135abf9f959_isrForCrosstalkSources_1123958_25.stderr parsl.dataflow.dflow INFO: Task 62 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2408ada73906cbb78a2ccec418ec5fb with result from task 62 parsl.dataflow.dflow DEBUG: Task 62 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/6515e6a8-194a-44cd-8306-a74d82222bed_isrForCrosstalkSources_1123958_56.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/6515e6a8-194a-44cd-8306-a74d82222bed_isrForCrosstalkSources_1123958_56.stderr parsl.dataflow.dflow INFO: Task 119 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3df9830e35f6b184f3761453684fe653 with result from task 119 parsl.dataflow.dflow DEBUG: Task 119 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/7ef0ee61-f593-4a64-bf66-682c439f279e_isrForCrosstalkSources_1123954_31.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/7ef0ee61-f593-4a64-bf66-682c439f279e_isrForCrosstalkSources_1123954_31.stderr parsl.dataflow.dflow INFO: Task 115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0685316270a68adccacdd7bf233cc848 with result from task 115 parsl.dataflow.dflow DEBUG: Task 115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/89d90e91-cf9c-4e2d-aa52-22dd85df9547_isrForCrosstalkSources_1123956_2.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/89d90e91-cf9c-4e2d-aa52-22dd85df9547_isrForCrosstalkSources_1123956_2.stderr parsl.dataflow.dflow INFO: Task 117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71a064758331336aef566f03765d03a2 with result from task 117 parsl.dataflow.dflow DEBUG: Task 117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/f87f47fa-b8f7-4ff4-99f1-3c31502c62fa_isrForCrosstalkSources_1123958_59.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/f87f47fa-b8f7-4ff4-99f1-3c31502c62fa_isrForCrosstalkSources_1123958_59.stderr parsl.dataflow.dflow INFO: Task 118 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3f8fb1ea2ce449f791975dbb09fbcf1 with result from task 118 parsl.dataflow.dflow DEBUG: Task 118 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/aa729c4d-f4c4-4192-b0d0-382188322e22_isrForCrosstalkSources_1123958_54.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/aa729c4d-f4c4-4192-b0d0-382188322e22_isrForCrosstalkSources_1123958_54.stderr parsl.dataflow.dflow INFO: Task 116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4bf1a51bbc21ce08c4ac283134498b9 with result from task 116 parsl.dataflow.dflow DEBUG: Task 116 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/d9a4ffa4-8714-48e9-9a97-aa37e412b757_isrForCrosstalkSources_1123956_34.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/d9a4ffa4-8714-48e9-9a97-aa37e412b757_isrForCrosstalkSources_1123956_34.stderr parsl.dataflow.dflow INFO: Task 122 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8e75b4b535dc28c4b43cc5aa8d3de93 with result from task 122 parsl.dataflow.dflow DEBUG: Task 122 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/ea9e6cf1-594e-489c-b122-34d1bad1e7c7_isrForCrosstalkSources_1123956_56.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/ea9e6cf1-594e-489c-b122-34d1bad1e7c7_isrForCrosstalkSources_1123956_56.stderr parsl.dataflow.dflow INFO: Task 121 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 927291f6b7fdc49110a550d433280fca with result from task 121 parsl.dataflow.dflow DEBUG: Task 121 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/793ffd1d-8f22-4a2a-bc0f-3df2de9af972_isrForCrosstalkSources_1123958_8.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/793ffd1d-8f22-4a2a-bc0f-3df2de9af972_isrForCrosstalkSources_1123958_8.stderr parsl.dataflow.dflow INFO: Task 175 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21295bfd0597173015695feac0df41b7 with result from task 175 parsl.dataflow.dflow DEBUG: Task 175 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/ee36f9bd-5e7e-48a4-95f1-b38aeec8f20a_isrForCrosstalkSources_1123956_10.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/ee36f9bd-5e7e-48a4-95f1-b38aeec8f20a_isrForCrosstalkSources_1123956_10.stderr parsl.dataflow.dflow INFO: Task 177 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6180d0f0fffe977ad2e648f31e73550d with result from task 177 parsl.dataflow.dflow DEBUG: Task 177 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/bf2aebb3-70dc-4ee5-94d6-a3cf8372fc0d_isrForCrosstalkSources_1123954_7.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/bf2aebb3-70dc-4ee5-94d6-a3cf8372fc0d_isrForCrosstalkSources_1123954_7.stderr parsl.dataflow.dflow INFO: Task 174 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1dead3ecedc4d3b85fff208e4e07cc3f with result from task 174 parsl.dataflow.dflow DEBUG: Task 174 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/5875cd7a-a43c-4070-b518-72819133dfce_isrForCrosstalkSources_1123956_61.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/5875cd7a-a43c-4070-b518-72819133dfce_isrForCrosstalkSources_1123956_61.stderr parsl.dataflow.dflow INFO: Task 176 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 145344565f7dca68864dad336823257c with result from task 176 parsl.dataflow.dflow DEBUG: Task 176 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/06a65fda-76f8-4170-83a2-bc68462145e5_isrForCrosstalkSources_1123956_59.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/06a65fda-76f8-4170-83a2-bc68462145e5_isrForCrosstalkSources_1123956_59.stderr parsl.dataflow.dflow INFO: Task 193 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67dacfb52251eb0b96dd98e85a1a250a with result from task 193 parsl.dataflow.dflow DEBUG: Task 193 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/3b18fd6a-f0d1-4b6e-b05a-f6c877c7a402_isrForCrosstalkSources_1123958_21.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/3b18fd6a-f0d1-4b6e-b05a-f6c877c7a402_isrForCrosstalkSources_1123958_21.stderr parsl.dataflow.dflow INFO: Task 114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01f0499161e0d9b68291f6be8aa92835 with result from task 114 parsl.dataflow.dflow DEBUG: Task 114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/4f46ce8c-b0d7-450e-9a35-62e9d291ee4e_isrForCrosstalkSources_1123958_49.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/4f46ce8c-b0d7-450e-9a35-62e9d291ee4e_isrForCrosstalkSources_1123958_49.stderr parsl.dataflow.dflow INFO: Task 112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d89e4bdd3837f908833a80f3b2f0035f with result from task 112 parsl.dataflow.dflow DEBUG: Task 112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/872fafd6-890f-43b7-9591-a9de07a728f6_isrForCrosstalkSources_1123956_43.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/872fafd6-890f-43b7-9591-a9de07a728f6_isrForCrosstalkSources_1123956_43.stderr parsl.dataflow.dflow INFO: Task 109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c8c9f0a5b2c3fbb085c3b8524a87aeb with result from task 109 parsl.dataflow.dflow DEBUG: Task 109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/dfe134bd-4d3f-4922-937a-009cc0a9b783_isrForCrosstalkSources_1123954_21.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/dfe134bd-4d3f-4922-937a-009cc0a9b783_isrForCrosstalkSources_1123954_21.stderr parsl.dataflow.dflow INFO: Task 111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa7f5a2911b0642677b4f9f2947bc67d with result from task 111 parsl.dataflow.dflow DEBUG: Task 111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/0ad32afc-f447-443f-a6bc-946d99e410d3_isrForCrosstalkSources_1123956_58.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/0ad32afc-f447-443f-a6bc-946d99e410d3_isrForCrosstalkSources_1123956_58.stderr parsl.dataflow.dflow INFO: Task 198 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 800f92d7171efac0c5433de0ead923ba with result from task 198 parsl.dataflow.dflow DEBUG: Task 198 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/d297b500-bf99-483c-bd86-32e3981ec974_isrForCrosstalkSources_1123954_25.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/d297b500-bf99-483c-bd86-32e3981ec974_isrForCrosstalkSources_1123954_25.stderr parsl.dataflow.dflow INFO: Task 191 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 154b9d7c831ec00b898b40604fdfa6df with result from task 191 parsl.dataflow.dflow DEBUG: Task 191 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/9ce31ac4-4d09-4644-b2ef-66e15db0af37_isrForCrosstalkSources_1123958_40.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/9ce31ac4-4d09-4644-b2ef-66e15db0af37_isrForCrosstalkSources_1123958_40.stderr parsl.dataflow.dflow INFO: Task 190 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d24f3053cafdb82c043e1facfff0a8a with result from task 190 parsl.dataflow.dflow DEBUG: Task 190 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/51f10e95-5757-4ea4-9a53-1deed5b2164a_isrForCrosstalkSources_1123958_29.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/51f10e95-5757-4ea4-9a53-1deed5b2164a_isrForCrosstalkSources_1123958_29.stderr parsl.dataflow.dflow INFO: Task 182 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc3a8d647973242d0919e2a2fa10fa4e with result from task 182 parsl.dataflow.dflow DEBUG: Task 182 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/2ee915de-2511-4d22-af35-6ff774e6b256_isrForCrosstalkSources_1123956_57.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/2ee915de-2511-4d22-af35-6ff774e6b256_isrForCrosstalkSources_1123956_57.stderr parsl.dataflow.dflow INFO: Task 178 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e624951a502924ae28b41d9ba00e8358 with result from task 178 parsl.dataflow.dflow DEBUG: Task 178 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/69518b60-790c-40a5-ba10-165a89a77625_isrForCrosstalkSources_1124087_9.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/69518b60-790c-40a5-ba10-165a89a77625_isrForCrosstalkSources_1124087_9.stderr parsl.dataflow.dflow INFO: Task 113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca95c99f264313141a07e9c983722c84 with result from task 113 parsl.dataflow.dflow DEBUG: Task 113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/54f7a794-a96b-4260-b984-65cb9bd6f1b9_isrForCrosstalkSources_1124087_22.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/54f7a794-a96b-4260-b984-65cb9bd6f1b9_isrForCrosstalkSources_1124087_22.stderr parsl.dataflow.dflow INFO: Task 110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4277506166d5fde3b5a6433df536cd6 with result from task 110 parsl.dataflow.dflow DEBUG: Task 110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/14966996-bc37-467d-bccf-66c6127e28de_isrForCrosstalkSources_1124087_29.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/14966996-bc37-467d-bccf-66c6127e28de_isrForCrosstalkSources_1124087_29.stderr parsl.dataflow.dflow INFO: Task 184 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9fb5def48528249b95dd9ed15724e39 with result from task 184 parsl.dataflow.dflow DEBUG: Task 184 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/19ce9793-ad83-4eb2-8451-6812d0bf21e0_isrForCrosstalkSources_1123954_8.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/19ce9793-ad83-4eb2-8451-6812d0bf21e0_isrForCrosstalkSources_1123954_8.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 115 parsl.dataflow.strategy DEBUG: Executor multi has 115 active tasks, 61/1 running/pending blocks, and 97 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22988724935824 parsl.dataflow.dflow INFO: Task 186 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 327e9513e29ef0dc57f944d435c5d44c with result from task 186 parsl.dataflow.dflow DEBUG: Task 186 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/4cf28a67-fc6c-4a59-838f-3234e462a29d_isrForCrosstalkSources_1123956_51.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/4cf28a67-fc6c-4a59-838f-3234e462a29d_isrForCrosstalkSources_1123956_51.stderr parsl.dataflow.dflow INFO: Task 173 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a50e59004db91a56eb0828d17c603562 with result from task 173 parsl.dataflow.dflow DEBUG: Task 173 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/79f6a555-7b9c-45e9-8e3c-0fe865f5b3fc_isrForCrosstalkSources_1124087_38.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/79f6a555-7b9c-45e9-8e3c-0fe865f5b3fc_isrForCrosstalkSources_1124087_38.stderr parsl.dataflow.dflow INFO: Task 120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 854aa1d50e3242caf0895d2c69813ce6 with result from task 120 parsl.dataflow.dflow DEBUG: Task 120 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/ed9fe296-3ee6-42cc-8c2a-2246a1fa8645_isrForCrosstalkSources_1124087_57.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/ed9fe296-3ee6-42cc-8c2a-2246a1fa8645_isrForCrosstalkSources_1124087_57.stderr parsl.dataflow.dflow INFO: Task 187 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c215221b0862892b49390ecbbc805fa6 with result from task 187 parsl.dataflow.dflow DEBUG: Task 187 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/b93ca3b6-8fbe-4b6b-ac17-182185d759f1_isrForCrosstalkSources_1123954_17.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/b93ca3b6-8fbe-4b6b-ac17-182185d759f1_isrForCrosstalkSources_1123954_17.stderr parsl.dataflow.dflow INFO: Task 108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b995076345343a245aebb98b128b69e0 with result from task 108 parsl.dataflow.dflow DEBUG: Task 108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/37ff3efe-5e02-4569-a24f-c94b8ead2772_isrForCrosstalkSources_1124087_1.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/37ff3efe-5e02-4569-a24f-c94b8ead2772_isrForCrosstalkSources_1124087_1.stderr parsl.dataflow.dflow INFO: Task 107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d705b5e1a45d887c0f7e503cdc406af9 with result from task 107 parsl.dataflow.dflow DEBUG: Task 107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/d720f84f-c9c1-4437-ae48-c9af7d7b62da_isrForCrosstalkSources_1124087_32.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/d720f84f-c9c1-4437-ae48-c9af7d7b62da_isrForCrosstalkSources_1124087_32.stderr parsl.dataflow.dflow INFO: Task 188 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9b84c4b2a7c5a2c2a11a70709fef6b6 with result from task 188 parsl.dataflow.dflow DEBUG: Task 188 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/3e454e00-9965-48ad-97e4-718f97943002_isrForCrosstalkSources_1124087_61.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/3e454e00-9965-48ad-97e4-718f97943002_isrForCrosstalkSources_1124087_61.stderr parsl.dataflow.dflow INFO: Task 189 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 508a9cee90a9b0ce22e5c718c1a42643 with result from task 189 parsl.dataflow.dflow DEBUG: Task 189 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/afa3fcd5-c210-4377-a16f-2aecb6406197_isrForCrosstalkSources_1123958_19.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/afa3fcd5-c210-4377-a16f-2aecb6406197_isrForCrosstalkSources_1123958_19.stderr parsl.dataflow.dflow INFO: Task 179 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 916f494e6102a0b953c64e9de58b64f1 with result from task 179 parsl.dataflow.dflow DEBUG: Task 179 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/b6dd5e9f-c2d0-49b9-8078-00b3ff7ddb74_isrForCrosstalkSources_1123958_62.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/b6dd5e9f-c2d0-49b9-8078-00b3ff7ddb74_isrForCrosstalkSources_1123958_62.stderr parsl.dataflow.dflow INFO: Task 127 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa1d3a268c02bedb9a22bda07663e77c with result from task 127 parsl.dataflow.dflow DEBUG: Task 127 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/1e94c2b4-71bb-4d3f-98ef-ebcfb0ec4eef_isrForCrosstalkSources_1123956_39.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/1e94c2b4-71bb-4d3f-98ef-ebcfb0ec4eef_isrForCrosstalkSources_1123956_39.stderr parsl.dataflow.dflow INFO: Task 126 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a557a69cdd2a4d83f4569beac843acfa with result from task 126 parsl.dataflow.dflow DEBUG: Task 126 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/d3326318-82c6-4356-8b17-c4546fea4009_isrForCrosstalkSources_1123956_24.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/d3326318-82c6-4356-8b17-c4546fea4009_isrForCrosstalkSources_1123956_24.stderr parsl.dataflow.dflow INFO: Task 125 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08c5f7288f38035349cdb29a01aa2518 with result from task 125 parsl.dataflow.dflow DEBUG: Task 125 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/227bc79f-5f0f-46d5-83cc-a6ff837eee3c_isrForCrosstalkSources_1123954_2.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/227bc79f-5f0f-46d5-83cc-a6ff837eee3c_isrForCrosstalkSources_1123954_2.stderr parsl.dataflow.dflow INFO: Task 124 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec53670fe5b2426b755f3bc208079c83 with result from task 124 parsl.dataflow.dflow DEBUG: Task 124 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/54f56583-d516-4d3f-a222-1e421ea648db_isrForCrosstalkSources_1123954_54.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/54f56583-d516-4d3f-a222-1e421ea648db_isrForCrosstalkSources_1123954_54.stderr parsl.dataflow.dflow INFO: Task 130 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61e2e79cf21c8d3b2dc8ac40ffbbb408 with result from task 130 parsl.dataflow.dflow DEBUG: Task 130 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/d35e5ca1-a1ae-4e69-aeb1-84891ae4bb75_isrForCrosstalkSources_1123954_61.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/d35e5ca1-a1ae-4e69-aeb1-84891ae4bb75_isrForCrosstalkSources_1123954_61.stderr parsl.dataflow.dflow INFO: Task 185 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51227af4c82fa2fce813755ef7fc0b47 with result from task 185 parsl.dataflow.dflow DEBUG: Task 185 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/6bbea2a1-da77-4756-b6cc-6a7c19d434ab_isrForCrosstalkSources_1123956_49.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/6bbea2a1-da77-4756-b6cc-6a7c19d434ab_isrForCrosstalkSources_1123956_49.stderr parsl.dataflow.dflow INFO: Task 194 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c84b911ed8a24950ecddce9382aa557 with result from task 194 parsl.dataflow.dflow DEBUG: Task 194 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/86543f81-6ec7-40c5-8cc1-897dc4eb2339_isrForCrosstalkSources_1123958_3.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/86543f81-6ec7-40c5-8cc1-897dc4eb2339_isrForCrosstalkSources_1123958_3.stderr parsl.dataflow.dflow INFO: Task 123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96c46b39b4d58ced056ba53a3886f1ea with result from task 123 parsl.dataflow.dflow DEBUG: Task 123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/b91f811c-50c3-4cf7-9531-52b55cfcc502_isrForCrosstalkSources_1123954_30.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/b91f811c-50c3-4cf7-9531-52b55cfcc502_isrForCrosstalkSources_1123954_30.stderr parsl.dataflow.dflow INFO: Task 128 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc81911324debd4921ad034839333a24 with result from task 128 parsl.dataflow.dflow DEBUG: Task 128 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/54eaecb4-80e5-4048-87d0-6b314831129c_isrForCrosstalkSources_1123954_60.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/54eaecb4-80e5-4048-87d0-6b314831129c_isrForCrosstalkSources_1123954_60.stderr parsl.dataflow.dflow INFO: Task 183 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 072c20f5a1c89a5db6fbef6e88d174de with result from task 183 parsl.dataflow.dflow DEBUG: Task 183 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/a50f25c0-e587-4077-a5f8-3076b5133993_isrForCrosstalkSources_1124087_36.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/a50f25c0-e587-4077-a5f8-3076b5133993_isrForCrosstalkSources_1124087_36.stderr parsl.dataflow.dflow INFO: Task 180 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1059fdfd79983ed3347a503881ac0833 with result from task 180 parsl.dataflow.dflow DEBUG: Task 180 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/9c1eba33-6446-4ff9-8e8e-a390300a2d7b_isrForCrosstalkSources_1124087_26.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/9c1eba33-6446-4ff9-8e8e-a390300a2d7b_isrForCrosstalkSources_1124087_26.stderr parsl.dataflow.dflow INFO: Task 181 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ddc3a3de14e22bbffcd5ff4cf24c8a2e with result from task 181 parsl.dataflow.dflow DEBUG: Task 181 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/fa9c00b5-c2dc-4c99-833e-2c489f5f8668_isrForCrosstalkSources_1124087_53.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/fa9c00b5-c2dc-4c99-833e-2c489f5f8668_isrForCrosstalkSources_1124087_53.stderr parsl.dataflow.dflow INFO: Task 196 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9e5023137e11ee0c79707b9533d6869 with result from task 196 parsl.dataflow.dflow DEBUG: Task 196 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/16592b77-0e74-458c-b0ae-816c226f9d1e_isrForCrosstalkSources_1123958_52.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/16592b77-0e74-458c-b0ae-816c226f9d1e_isrForCrosstalkSources_1123958_52.stderr parsl.dataflow.dflow INFO: Task 136 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0eaefa6f45893698ad0b8a3dbce06af with result from task 136 parsl.dataflow.dflow DEBUG: Task 136 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/56444a86-f1bb-49f7-b5a7-52bc0bea8f47_isrForCrosstalkSources_1124087_59.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/56444a86-f1bb-49f7-b5a7-52bc0bea8f47_isrForCrosstalkSources_1124087_59.stderr parsl.dataflow.dflow INFO: Task 138 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6bc1d32c0151d8b92f8420ad87b7a471 with result from task 138 parsl.dataflow.dflow DEBUG: Task 138 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/0a1efa06-1e15-4a67-8cb4-22a05de1d5df_isrForCrosstalkSources_1123956_27.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/0a1efa06-1e15-4a67-8cb4-22a05de1d5df_isrForCrosstalkSources_1123956_27.stderr parsl.dataflow.dflow INFO: Task 134 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd39e71bdb3943502b785c2a4b163eb7 with result from task 134 parsl.dataflow.dflow DEBUG: Task 134 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/44acff89-0507-4c2f-88db-1eddd0f46f85_isrForCrosstalkSources_1123958_20.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/44acff89-0507-4c2f-88db-1eddd0f46f85_isrForCrosstalkSources_1123958_20.stderr parsl.dataflow.dflow INFO: Task 131 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c92f5905a721f3ac17c1391c97d51b4 with result from task 131 parsl.dataflow.dflow DEBUG: Task 131 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/6d7f8559-b8e6-47ef-8c66-9a625019b568_isrForCrosstalkSources_1124087_48.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/6d7f8559-b8e6-47ef-8c66-9a625019b568_isrForCrosstalkSources_1124087_48.stderr parsl.dataflow.dflow INFO: Task 135 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 301cb286aefb4baf6e6f0396f87ef9a9 with result from task 135 parsl.dataflow.dflow DEBUG: Task 135 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/ce701303-d53c-43d5-a717-a61e9449da37_isrForCrosstalkSources_1123956_53.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/ce701303-d53c-43d5-a717-a61e9449da37_isrForCrosstalkSources_1123956_53.stderr parsl.dataflow.dflow INFO: Task 132 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51699c7daa31e33728818cd36acf8000 with result from task 132 parsl.dataflow.dflow DEBUG: Task 132 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/3cbb1ba1-c09a-4f8a-bef2-e39b3a35d2a7_isrForCrosstalkSources_1124087_18.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/3cbb1ba1-c09a-4f8a-bef2-e39b3a35d2a7_isrForCrosstalkSources_1124087_18.stderr parsl.dataflow.dflow INFO: Task 133 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f373b0cab3b71ac394c8adeb97d275f5 with result from task 133 parsl.dataflow.dflow DEBUG: Task 133 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/57ebdb48-8f3c-4ca2-8fb0-da0f71c195e4_isrForCrosstalkSources_1123958_2.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/57ebdb48-8f3c-4ca2-8fb0-da0f71c195e4_isrForCrosstalkSources_1123958_2.stderr parsl.dataflow.dflow INFO: Task 137 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry affa0b18207738fc6e1fa324bff82bbf with result from task 137 parsl.dataflow.dflow DEBUG: Task 137 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/88566225-38b2-4096-825f-b65e7ca1ae45_isrForCrosstalkSources_1123956_17.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/88566225-38b2-4096-825f-b65e7ca1ae45_isrForCrosstalkSources_1123956_17.stderr parsl.dataflow.dflow INFO: Task 129 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a35ecceea021a536430b90579a1fa65 with result from task 129 parsl.dataflow.dflow DEBUG: Task 129 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/b1d4f3c9-db0e-425d-97b1-031c9cbfd1de_isrForCrosstalkSources_1124087_20.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/b1d4f3c9-db0e-425d-97b1-031c9cbfd1de_isrForCrosstalkSources_1124087_20.stderr parsl.dataflow.dflow INFO: Task 199 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 172bd129de1833e8e8f82dc3881a6a99 with result from task 199 parsl.dataflow.dflow DEBUG: Task 199 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/3e386935-b586-497b-ad94-1cae405043a2_isrForCrosstalkSources_1123954_56.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/3e386935-b586-497b-ad94-1cae405043a2_isrForCrosstalkSources_1123954_56.stderr parsl.dataflow.dflow INFO: Task 197 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d75c68f195ef9b4de5802c32f3528606 with result from task 197 parsl.dataflow.dflow DEBUG: Task 197 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/5084079e-67e5-4a28-8e6c-331dd2451632_isrForCrosstalkSources_1123954_59.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/5084079e-67e5-4a28-8e6c-331dd2451632_isrForCrosstalkSources_1123954_59.stderr parsl.dataflow.dflow INFO: Task 192 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d4b165a15839bc58fdbbab2a9873a11 with result from task 192 parsl.dataflow.dflow DEBUG: Task 192 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/867ddbde-dad0-4d91-8719-8c567657a4d7_isrForCrosstalkSources_1124087_34.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/867ddbde-dad0-4d91-8719-8c567657a4d7_isrForCrosstalkSources_1124087_34.stderr parsl.dataflow.dflow INFO: Task 202 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb6163e3968e5182fbd3eb4e3e8f753f with result from task 202 parsl.dataflow.dflow DEBUG: Task 202 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/f18f6540-5cdf-4599-a087-ca00a82e368c_isrForCrosstalkSources_1123958_41.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/f18f6540-5cdf-4599-a087-ca00a82e368c_isrForCrosstalkSources_1123958_41.stderr parsl.dataflow.dflow INFO: Task 203 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d970a1d873a940d204659bc32ed07fe with result from task 203 parsl.dataflow.dflow DEBUG: Task 203 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/799e5421-61c6-4b86-9ad1-a22582b8a548_isrForCrosstalkSources_1123958_60.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/799e5421-61c6-4b86-9ad1-a22582b8a548_isrForCrosstalkSources_1123958_60.stderr parsl.dataflow.dflow INFO: Task 201 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 478af14e4fa9af6397f741c3bb7a3fc4 with result from task 201 parsl.dataflow.dflow DEBUG: Task 201 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/4f13638b-8080-4e43-a286-cd9b0065b760_isrForCrosstalkSources_1123958_31.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/4f13638b-8080-4e43-a286-cd9b0065b760_isrForCrosstalkSources_1123958_31.stderr parsl.dataflow.dflow INFO: Task 195 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ceb50c6c21265b054f3c40d208fa624a with result from task 195 parsl.dataflow.dflow DEBUG: Task 195 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/f3e48781-ad00-44eb-ae06-a34171c071ee_isrForCrosstalkSources_1124087_60.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/f3e48781-ad00-44eb-ae06-a34171c071ee_isrForCrosstalkSources_1124087_60.stderr parsl.dataflow.dflow INFO: Task 200 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad3a53a6373e83d7e8157f05d9a663ab with result from task 200 parsl.dataflow.dflow DEBUG: Task 200 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/3d39beea-21e2-4d7e-876b-8bc04c95998d_isrForCrosstalkSources_1124087_52.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/3d39beea-21e2-4d7e-876b-8bc04c95998d_isrForCrosstalkSources_1124087_52.stderr 2024-03-07 03:35:33 proc_lsst.multi:146 [INFO] found job 766 in provider local proc_lsst.multi INFO: found job 766 in provider local 2024-03-07 03:35:33 proc_lsst.multi:146 [INFO] found job 16884172 in provider astro proc_lsst.multi INFO: found job 16884172 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:34 proc_lsst.multi:146 [INFO] found job 16884173 in provider astro proc_lsst.multi INFO: found job 16884173 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:34 proc_lsst.multi:146 [INFO] found job 16884174 in provider astro proc_lsst.multi INFO: found job 16884174 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:34 proc_lsst.multi:146 [INFO] found job 16884175 in provider astro proc_lsst.multi INFO: found job 16884175 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.dataflow.dflow INFO: Task 143 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23964519869feefe88cf526c52652254 with result from task 143 parsl.dataflow.dflow DEBUG: Task 143 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/6d3a7f42-3a14-4cab-8b2b-abb1ae3119ac_isrForCrosstalkSources_1123958_38.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/6d3a7f42-3a14-4cab-8b2b-abb1ae3119ac_isrForCrosstalkSources_1123958_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:35 proc_lsst.multi:146 [INFO] found job 16884176 in provider astro proc_lsst.multi INFO: found job 16884176 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:35 proc_lsst.multi:146 [INFO] found job 16884177 in provider astro proc_lsst.multi INFO: found job 16884177 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:35 proc_lsst.multi:146 [INFO] found job 16884178 in provider astro proc_lsst.multi INFO: found job 16884178 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:35 proc_lsst.multi:146 [INFO] found job 16884179 in provider astro proc_lsst.multi INFO: found job 16884179 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.dataflow.dflow INFO: Task 222 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f42706d190b3414c362a6d9138901f17 with result from task 222 parsl.dataflow.dflow DEBUG: Task 222 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/fbbf51cb-65b5-4c12-bf45-897c556694f7_isrForCrosstalkSources_1123954_58.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/fbbf51cb-65b5-4c12-bf45-897c556694f7_isrForCrosstalkSources_1123954_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:35 proc_lsst.multi:146 [INFO] found job 16884180 in provider astro proc_lsst.multi INFO: found job 16884180 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:36 proc_lsst.multi:146 [INFO] found job 16884181 in provider astro proc_lsst.multi INFO: found job 16884181 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.dataflow.dflow INFO: Task 217 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 991732ca8fb5e8205673905aff276d3c with result from task 217 parsl.dataflow.dflow DEBUG: Task 217 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/1334ee97-84bb-4ffc-be56-c5f45b9d9bd8_isrForCrosstalkSources_1123956_48.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/1334ee97-84bb-4ffc-be56-c5f45b9d9bd8_isrForCrosstalkSources_1123956_48.stderr parsl.dataflow.dflow INFO: Task 206 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a676417a7ee2511286bce71f920cac7b with result from task 206 parsl.dataflow.dflow DEBUG: Task 206 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/48355fd9-0bc7-40af-acc5-67d82003d614_isrForCrosstalkSources_1123958_42.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/48355fd9-0bc7-40af-acc5-67d82003d614_isrForCrosstalkSources_1123958_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:36 proc_lsst.multi:146 [INFO] found job 16884182 in provider astro proc_lsst.multi INFO: found job 16884182 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.dataflow.dflow INFO: Task 207 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 294293750313be28a8523109830170b2 with result from task 207 parsl.dataflow.dflow DEBUG: Task 207 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/b4b1e2fc-82f4-4e3d-bd21-d94ac07b818c_isrForCrosstalkSources_1123954_28.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/b4b1e2fc-82f4-4e3d-bd21-d94ac07b818c_isrForCrosstalkSources_1123954_28.stderr parsl.dataflow.dflow INFO: Task 212 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd73e99871d6ea1b5063cbc655b6d872 with result from task 212 parsl.dataflow.dflow DEBUG: Task 212 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/03194941-efbf-4417-8dff-df1208ffecfd_isrForCrosstalkSources_1123956_9.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/03194941-efbf-4417-8dff-df1208ffecfd_isrForCrosstalkSources_1123956_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:36 proc_lsst.multi:146 [INFO] found job 16884183 in provider astro proc_lsst.multi INFO: found job 16884183 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:36 proc_lsst.multi:146 [INFO] found job 16884184 in provider astro proc_lsst.multi INFO: found job 16884184 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.dataflow.dflow INFO: Task 208 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 679ce072931344c508a595989f996436 with result from task 208 parsl.dataflow.dflow DEBUG: Task 208 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/8bf2b414-3e56-41be-bf93-8edb5eddb992_isrForCrosstalkSources_1123958_37.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/8bf2b414-3e56-41be-bf93-8edb5eddb992_isrForCrosstalkSources_1123958_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:36 proc_lsst.multi:146 [INFO] found job 16884185 in provider astro proc_lsst.multi INFO: found job 16884185 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.dataflow.dflow INFO: Task 205 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 680134c621504df9c24290d5092f7f55 with result from task 205 parsl.dataflow.dflow DEBUG: Task 205 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/8c09b022-90e6-4b21-b6a9-226dd17e5471_isrForCrosstalkSources_1123956_14.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/8c09b022-90e6-4b21-b6a9-226dd17e5471_isrForCrosstalkSources_1123956_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:36 proc_lsst.multi:146 [INFO] found job 16884186 in provider astro proc_lsst.multi INFO: found job 16884186 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.dataflow.dflow INFO: Task 210 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d4ad50cfdbc3745b7ae83a27e609998 with result from task 210 parsl.dataflow.dflow DEBUG: Task 210 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/5e2e2e05-a61e-4c94-aa28-b208cf7deef7_isrForCrosstalkSources_1123954_49.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/5e2e2e05-a61e-4c94-aa28-b208cf7deef7_isrForCrosstalkSources_1123954_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:36 proc_lsst.multi:146 [INFO] found job 16884187 in provider astro proc_lsst.multi INFO: found job 16884187 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:36 proc_lsst.multi:146 [INFO] found job 16884188 in provider astro proc_lsst.multi INFO: found job 16884188 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:37 proc_lsst.multi:146 [INFO] found job 16884189 in provider astro proc_lsst.multi INFO: found job 16884189 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.dataflow.dflow INFO: Task 214 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 785c8b0588283672dc1e22b85278f007 with result from task 214 parsl.dataflow.dflow DEBUG: Task 214 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/3827a433-3b5a-4199-acd8-4e3b77d1c0e4_isrForCrosstalkSources_1123956_46.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/3827a433-3b5a-4199-acd8-4e3b77d1c0e4_isrForCrosstalkSources_1123956_46.stderr parsl.dataflow.dflow INFO: Task 215 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12334ec4c541b35b010476625a72e188 with result from task 215 parsl.dataflow.dflow DEBUG: Task 215 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/dcb3fb92-2730-4537-ba80-49611876fdca_isrForCrosstalkSources_1123954_44.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/dcb3fb92-2730-4537-ba80-49611876fdca_isrForCrosstalkSources_1123954_44.stderr parsl.dataflow.dflow INFO: Task 204 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59bd213208706378ece6757b18374b49 with result from task 204 parsl.dataflow.dflow DEBUG: Task 204 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/813e9cd0-a9c5-4379-9afd-fd3ad32257b0_isrForCrosstalkSources_1123954_29.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/813e9cd0-a9c5-4379-9afd-fd3ad32257b0_isrForCrosstalkSources_1123954_29.stderr parsl.dataflow.dflow INFO: Task 216 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ca69430d9708157dfe9675cb4eda1b2 with result from task 216 parsl.dataflow.dflow DEBUG: Task 216 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/0978e9e6-ea35-4e08-9490-983681677f7a_isrForCrosstalkSources_1123958_15.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/0978e9e6-ea35-4e08-9490-983681677f7a_isrForCrosstalkSources_1123958_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:37 proc_lsst.multi:146 [INFO] found job 16884190 in provider astro proc_lsst.multi INFO: found job 16884190 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:37 proc_lsst.multi:146 [INFO] found job 16884191 in provider astro proc_lsst.multi INFO: found job 16884191 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:37 proc_lsst.multi:146 [INFO] found job 16884192 in provider astro proc_lsst.multi INFO: found job 16884192 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.dataflow.dflow INFO: Task 223 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58e48215da2cf8e9570058f6bcc1b2e6 with result from task 223 parsl.dataflow.dflow DEBUG: Task 223 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/eb005449-2d24-41b2-9e1a-4dc0f0bf15fe_isrForCrosstalkSources_1123956_12.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/eb005449-2d24-41b2-9e1a-4dc0f0bf15fe_isrForCrosstalkSources_1123956_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:37 proc_lsst.multi:146 [INFO] found job 16884193 in provider astro proc_lsst.multi INFO: found job 16884193 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:37 proc_lsst.multi:146 [INFO] found job 16884194 in provider astro proc_lsst.multi INFO: found job 16884194 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:37 proc_lsst.multi:146 [INFO] found job 16884195 in provider astro proc_lsst.multi INFO: found job 16884195 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:37 proc_lsst.multi:146 [INFO] found job 16884196 in provider astro proc_lsst.multi INFO: found job 16884196 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:38 proc_lsst.multi:146 [INFO] found job 16884197 in provider astro proc_lsst.multi INFO: found job 16884197 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:38 proc_lsst.multi:146 [INFO] found job 16884203 in provider astro proc_lsst.multi INFO: found job 16884203 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:38 proc_lsst.multi:146 [INFO] found job 16884204 in provider astro proc_lsst.multi INFO: found job 16884204 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.dataflow.dflow INFO: Task 226 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c25967c623ed784a46bfb834577c2a18 with result from task 226 parsl.dataflow.dflow DEBUG: Task 226 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/e83904b9-f899-417b-b6c7-511e6dd7d9b8_isrForCrosstalkSources_1123956_60.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/e83904b9-f899-417b-b6c7-511e6dd7d9b8_isrForCrosstalkSources_1123956_60.stderr parsl.dataflow.dflow INFO: Task 231 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30ac6968a39f214c30413c7ae66864e3 with result from task 231 parsl.dataflow.dflow DEBUG: Task 231 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/16654c15-77c3-46d2-af84-a48e6b76433a_isrForCrosstalkSources_1123956_8.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/16654c15-77c3-46d2-af84-a48e6b76433a_isrForCrosstalkSources_1123956_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:38 proc_lsst.multi:146 [INFO] found job 16884205 in provider astro proc_lsst.multi INFO: found job 16884205 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.dataflow.dflow INFO: Task 224 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a039309851f3397b60bc852e74067dbf with result from task 224 parsl.dataflow.dflow DEBUG: Task 224 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/e70a4d40-d3f0-4b9c-9b58-ee7e9b5b34b0_isrForCrosstalkSources_1123958_6.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/e70a4d40-d3f0-4b9c-9b58-ee7e9b5b34b0_isrForCrosstalkSources_1123958_6.stderr parsl.dataflow.dflow INFO: Task 225 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 970bfdaf30935549f3bfa36209416b9f with result from task 225 parsl.dataflow.dflow DEBUG: Task 225 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/e5a36366-cab6-4e95-ab00-8ff3c50e98e9_isrForCrosstalkSources_1123958_35.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/e5a36366-cab6-4e95-ab00-8ff3c50e98e9_isrForCrosstalkSources_1123958_35.stderr parsl.dataflow.dflow INFO: Task 161 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 305474b70f56ab7ac8c79215fbb77b6d with result from task 161 parsl.dataflow.dflow DEBUG: Task 161 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/e56b7125-1c6f-47ef-ae79-1e88744837a2_isrForCrosstalkSources_1123956_3.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/e56b7125-1c6f-47ef-ae79-1e88744837a2_isrForCrosstalkSources_1123956_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:38 proc_lsst.multi:146 [INFO] found job 16884206 in provider astro proc_lsst.multi INFO: found job 16884206 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.dataflow.dflow INFO: Task 148 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58350c5dcc6ebe1632f7601d7fd14130 with result from task 148 parsl.dataflow.dflow DEBUG: Task 148 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/32484b58-1e6b-4654-b317-cc927b6cb27f_isrForCrosstalkSources_1123956_55.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/32484b58-1e6b-4654-b317-cc927b6cb27f_isrForCrosstalkSources_1123956_55.stderr parsl.dataflow.dflow INFO: Task 220 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bfc011f73fdbf3fc67cb796c4c376668 with result from task 220 parsl.dataflow.dflow DEBUG: Task 220 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/b765faf0-acd4-429a-85e2-34db494106c8_isrForCrosstalkSources_1123956_52.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/b765faf0-acd4-429a-85e2-34db494106c8_isrForCrosstalkSources_1123956_52.stderr parsl.dataflow.dflow INFO: Task 150 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 030a2c9f0a51b285de92ad4fb444b1cb with result from task 150 parsl.dataflow.dflow DEBUG: Task 150 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/a26795b2-e3f9-414d-8a11-ef51f738e4a3_isrForCrosstalkSources_1123956_22.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/a26795b2-e3f9-414d-8a11-ef51f738e4a3_isrForCrosstalkSources_1123956_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:38 proc_lsst.multi:146 [INFO] found job 16884207 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884207 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884207,16884208,16884209,16884210,16884211' parsl.dataflow.dflow INFO: Task 236 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f55bc3614cea1f70570a75a73128200d with result from task 236 parsl.dataflow.dflow DEBUG: Task 236 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/cfb20153-8d2a-421d-892d-6d9cdfed28c4_isrForCrosstalkSources_1123954_26.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/cfb20153-8d2a-421d-892d-6d9cdfed28c4_isrForCrosstalkSources_1123954_26.stderr parsl.dataflow.dflow INFO: Task 238 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50aa0c2e2b64a39e27116c75d138dd00 with result from task 238 parsl.dataflow.dflow DEBUG: Task 238 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/1f99a5c0-b51a-477e-862a-f87901b0ebef_isrForCrosstalkSources_1123956_36.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/1f99a5c0-b51a-477e-862a-f87901b0ebef_isrForCrosstalkSources_1123956_36.stderr parsl.dataflow.dflow INFO: Task 239 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7af1bef951d9ba77d62835e0fbbd70e with result from task 239 parsl.dataflow.dflow DEBUG: Task 239 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/6dde717a-411b-4f04-91b0-05cce6a4b3ac_isrForCrosstalkSources_1123954_57.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/6dde717a-411b-4f04-91b0-05cce6a4b3ac_isrForCrosstalkSources_1123954_57.stderr parsl.dataflow.dflow INFO: Task 241 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: squeue returned 16884207 R 16884208 R 16884209 R 16884210 R 16884211 R parsl.dataflow.memoization DEBUG: Storing app cache entry f1f9eb9ea7da6b31328515b1966c28d9 with result from task 241 parsl.providers.slurm.slurm DEBUG: Updating job 16884207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884211 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:38 proc_lsst.multi:146 [INFO] found job 16884208 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884208 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884207,16884208,16884209,16884210,16884211' parsl.dataflow.dflow DEBUG: Task 241 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/bd236e00-9bca-4296-a3f4-b721ccabeb0b_isrForCrosstalkSources_1123956_28.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/bd236e00-9bca-4296-a3f4-b721ccabeb0b_isrForCrosstalkSources_1123956_28.stderr parsl.dataflow.dflow INFO: Task 240 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee47b7731e1fec21c197c0f4bd96fb53 with result from task 240 parsl.dataflow.dflow DEBUG: Task 240 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/9b0e094e-c248-4117-ac9d-6e23ad30ef0a_isrForCrosstalkSources_1123958_27.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/9b0e094e-c248-4117-ac9d-6e23ad30ef0a_isrForCrosstalkSources_1123958_27.stderr parsl.dataflow.dflow INFO: Task 211 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eac4b40767a07feabebc58754b1dc75b with result from task 211 parsl.dataflow.dflow DEBUG: Task 211 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/96dcbbfa-b490-4e0b-ae0e-e5dcee6d4b34_isrForCrosstalkSources_1124087_3.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/96dcbbfa-b490-4e0b-ae0e-e5dcee6d4b34_isrForCrosstalkSources_1124087_3.stderr parsl.dataflow.dflow INFO: Task 209 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80169c653a570c22b383752c99113c2e with result from task 209 parsl.dataflow.dflow DEBUG: Task 209 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/c281b501-0bce-4c32-9706-dccedb1111eb_isrForCrosstalkSources_1124087_13.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/c281b501-0bce-4c32-9706-dccedb1111eb_isrForCrosstalkSources_1124087_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884207 R 16884208 R 16884209 R 16884210 R 16884211 R parsl.providers.slurm.slurm DEBUG: Updating job 16884207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884211 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:39 proc_lsst.multi:146 [INFO] found job 16884209 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884209 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884207,16884208,16884209,16884210,16884211' parsl.providers.slurm.slurm DEBUG: squeue returned 16884207 R 16884208 R 16884209 R 16884210 R 16884211 R parsl.providers.slurm.slurm DEBUG: Updating job 16884207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884211 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:39 proc_lsst.multi:146 [INFO] found job 16884210 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884210 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884207,16884208,16884209,16884210,16884211' parsl.dataflow.dflow INFO: Task 230 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c786d170a9686b666956482315ba0e6 with result from task 230 parsl.dataflow.dflow DEBUG: Task 230 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/bf213554-e7f6-4647-8b58-45a4d9a9a368_isrForCrosstalkSources_1123958_47.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/bf213554-e7f6-4647-8b58-45a4d9a9a368_isrForCrosstalkSources_1123958_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884207 R 16884208 R 16884209 R 16884210 R 16884211 R parsl.providers.slurm.slurm DEBUG: Updating job 16884207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884211 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:39 proc_lsst.multi:146 [INFO] found job 16884211 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884211 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884207,16884208,16884209,16884210,16884211' parsl.dataflow.dflow INFO: Task 227 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f332a905b59259226f24766ab4c9d29 with result from task 227 parsl.dataflow.dflow DEBUG: Task 227 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/c1ec565a-773b-4689-a741-b8bc5bfc0b50_isrForCrosstalkSources_1123958_32.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/c1ec565a-773b-4689-a741-b8bc5bfc0b50_isrForCrosstalkSources_1123958_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884207 R 16884208 R 16884209 R 16884210 R 16884211 R parsl.providers.slurm.slurm DEBUG: Updating job 16884207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884211 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:39 proc_lsst.multi:146 [INFO] found job 16884212 in provider escience_a40 proc_lsst.multi INFO: found job 16884212 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.dataflow.dflow INFO: Task 244 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 674e343cd646b9e6bbf66984982a23c0 with result from task 244 parsl.dataflow.dflow DEBUG: Task 244 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/0251cf6d-dd64-479b-8b9a-edbd7169d519_isrForCrosstalkSources_1123956_41.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/0251cf6d-dd64-479b-8b9a-edbd7169d519_isrForCrosstalkSources_1123956_41.stderr parsl.dataflow.dflow INFO: Task 246 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9afa8adacf970c20b5d3d92f69a45cce with result from task 246 parsl.dataflow.dflow DEBUG: Task 246 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/d3414de2-7a4d-45b0-8f2f-f82781a8c644_isrForCrosstalkSources_1123958_16.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/d3414de2-7a4d-45b0-8f2f-f82781a8c644_isrForCrosstalkSources_1123958_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:39 proc_lsst.multi:146 [INFO] found job 16884213 in provider escience_a40 proc_lsst.multi INFO: found job 16884213 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.dataflow.dflow INFO: Task 235 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64a5ca4c7ff3bd2a49c09e0cf060f76a with result from task 235 parsl.dataflow.dflow DEBUG: Task 235 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/85ca26dc-37ab-4b03-ab76-a6da4ab961a5_isrForCrosstalkSources_1123954_40.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/85ca26dc-37ab-4b03-ab76-a6da4ab961a5_isrForCrosstalkSources_1123954_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:39 proc_lsst.multi:146 [INFO] found job 16884214 in provider escience_a40 proc_lsst.multi INFO: found job 16884214 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.dataflow.dflow INFO: Task 213 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c24b3ba9b21147bd10d534e4a6c4bc4 with result from task 213 parsl.dataflow.dflow DEBUG: Task 213 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/0548b533-9b78-4477-bc95-81408dfd3dfd_isrForCrosstalkSources_1124087_55.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/0548b533-9b78-4477-bc95-81408dfd3dfd_isrForCrosstalkSources_1124087_55.stderr parsl.dataflow.dflow INFO: Task 232 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53fbe425acf22a8f9fea030a3f57cbe6 with result from task 232 parsl.dataflow.dflow DEBUG: Task 232 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/64120a94-6f9f-443c-a831-6bb9c29165da_isrForCrosstalkSources_1124087_50.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/64120a94-6f9f-443c-a831-6bb9c29165da_isrForCrosstalkSources_1124087_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:39 proc_lsst.multi:146 [INFO] found job 16884215 in provider escience_a40 proc_lsst.multi INFO: found job 16884215 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:39 proc_lsst.multi:146 [INFO] found job 16884216 in provider escience_a40 proc_lsst.multi INFO: found job 16884216 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.dataflow.dflow INFO: Task 234 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97eee4528c737b04b402ae425a37cb8e with result from task 234 parsl.dataflow.dflow DEBUG: Task 234 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/3f2df7ee-b226-4fe8-b93e-251affc6888a_isrForCrosstalkSources_1123958_58.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/3f2df7ee-b226-4fe8-b93e-251affc6888a_isrForCrosstalkSources_1123958_58.stderr parsl.dataflow.dflow INFO: Task 229 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9fc24a1f784eb946ca49be1816f78ff with result from task 229 parsl.dataflow.dflow DEBUG: Task 229 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/2a9e3705-a9b3-458c-b32f-7eb0c9c6c962_isrForCrosstalkSources_1124087_49.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/2a9e3705-a9b3-458c-b32f-7eb0c9c6c962_isrForCrosstalkSources_1124087_49.stderr parsl.dataflow.dflow INFO: Task 247 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11ee8f98c333320c9acf3d79532138c4 with result from task 247 parsl.dataflow.dflow DEBUG: Task 247 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/8a73cfc3-7b7c-4e48-95d3-37e37d63d598_isrForCrosstalkSources_1123958_18.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/8a73cfc3-7b7c-4e48-95d3-37e37d63d598_isrForCrosstalkSources_1123958_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:40 proc_lsst.multi:146 [INFO] found job 16884217 in provider escience_a40 proc_lsst.multi INFO: found job 16884217 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.dataflow.dflow INFO: Task 243 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d6086b6d5ee38d3f0f0de63ad651e02d with result from task 243 parsl.dataflow.dflow DEBUG: Task 243 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/9f0639a0-0e87-4019-a70c-8b22f474df01_isrForCrosstalkSources_1124087_2.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/9f0639a0-0e87-4019-a70c-8b22f474df01_isrForCrosstalkSources_1124087_2.stderr parsl.dataflow.dflow INFO: Task 237 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94f78930f3e624689de7274f52c8ecf0 with result from task 237 parsl.dataflow.dflow DEBUG: Task 237 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/eeea1d88-dc49-4dbe-9da5-a42dcab5e9ba_isrForCrosstalkSources_1124087_31.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/eeea1d88-dc49-4dbe-9da5-a42dcab5e9ba_isrForCrosstalkSources_1124087_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:40 proc_lsst.multi:146 [INFO] found job 16884218 in provider escience_a40 proc_lsst.multi INFO: found job 16884218 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.dataflow.dflow INFO: Task 242 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70f7cfbb953f931c503be8dfbb4bffd6 with result from task 242 parsl.dataflow.dflow DEBUG: Task 242 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/abb236fd-8273-4d64-a7ce-65ed3b84b8a2_isrForCrosstalkSources_1124087_37.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/abb236fd-8273-4d64-a7ce-65ed3b84b8a2_isrForCrosstalkSources_1124087_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:40 proc_lsst.multi:146 [INFO] found job 16884219 in provider escience_a40 proc_lsst.multi INFO: found job 16884219 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:40 proc_lsst.multi:146 [INFO] found job 16884220 in provider escience_a40 proc_lsst.multi INFO: found job 16884220 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.dataflow.dflow INFO: Task 233 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd187def9aa7b12f3c1fc3a949c9d24e with result from task 233 parsl.dataflow.dflow DEBUG: Task 233 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/ea5d1c56-bbad-4220-b789-82785b2e269d_isrForCrosstalkSources_1124087_24.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/ea5d1c56-bbad-4220-b789-82785b2e269d_isrForCrosstalkSources_1124087_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:40 proc_lsst.multi:146 [INFO] found job 16884221 in provider escience_a40 proc_lsst.multi INFO: found job 16884221 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.dataflow.dflow INFO: Task 218 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7847dee8f9a178ca74b875d34421e447 with result from task 218 parsl.dataflow.dflow DEBUG: Task 218 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/a2699cca-e736-46ab-a92c-798a3f1cc841_isrForCrosstalkSources_1124087_41.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/a2699cca-e736-46ab-a92c-798a3f1cc841_isrForCrosstalkSources_1124087_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:40 proc_lsst.multi:146 [INFO] found job 16884222 in provider escience_a40 proc_lsst.multi INFO: found job 16884222 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.dataflow.dflow INFO: Task 219 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64e9ac0fc5ab0c550fe3edbe5a30c2f6 with result from task 219 parsl.dataflow.dflow DEBUG: Task 219 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/0b7c0643-7922-4ea4-b46f-ffca298258a0_isrForCrosstalkSources_1123958_51.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/0b7c0643-7922-4ea4-b46f-ffca298258a0_isrForCrosstalkSources_1123958_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:40 proc_lsst.multi:146 [INFO] found job 16884223 in provider escience_a40 proc_lsst.multi INFO: found job 16884223 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.dataflow.dflow INFO: Task 245 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4efea3eee8538f3d978d37dbedc66ffb with result from task 245 parsl.dataflow.dflow DEBUG: Task 245 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/81f808ef-9ef5-4c75-9015-4da76eb1302c_isrForCrosstalkSources_1124087_27.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/81f808ef-9ef5-4c75-9015-4da76eb1302c_isrForCrosstalkSources_1124087_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:40 proc_lsst.multi:146 [INFO] found job 16884224 in provider escience_a40 proc_lsst.multi INFO: found job 16884224 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.dataflow.dflow INFO: Task 228 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe5200417c9432cc7b0da372c4de0f28 with result from task 228 parsl.dataflow.dflow DEBUG: Task 228 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/294c06fa-fd04-44be-881d-c5ab168e7fd4_isrForCrosstalkSources_1124087_39.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/294c06fa-fd04-44be-881d-c5ab168e7fd4_isrForCrosstalkSources_1124087_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:41 proc_lsst.multi:146 [INFO] found job 16884225 in provider escience_a40 proc_lsst.multi INFO: found job 16884225 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.dataflow.dflow INFO: Task 221 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11330828fe9444165d1deff355b3bf1a with result from task 221 parsl.dataflow.dflow DEBUG: Task 221 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/b3450c1d-0001-4f93-a2ac-5498dcc86302_isrForCrosstalkSources_1123954_35.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/b3450c1d-0001-4f93-a2ac-5498dcc86302_isrForCrosstalkSources_1123954_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:41 proc_lsst.multi:146 [INFO] found job 16884226 in provider escience_a40 proc_lsst.multi INFO: found job 16884226 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:41 proc_lsst.multi:146 [INFO] found job 16884227 in provider escience_a40 proc_lsst.multi INFO: found job 16884227 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:41 proc_lsst.multi:146 [INFO] found job 16884228 in provider escience_a40 proc_lsst.multi INFO: found job 16884228 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:41 proc_lsst.multi:146 [INFO] found job 16884229 in provider escience_a40 proc_lsst.multi INFO: found job 16884229 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:41 proc_lsst.multi:146 [INFO] found job 16884230 in provider escience_a40 proc_lsst.multi INFO: found job 16884230 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:41 proc_lsst.multi:146 [INFO] found job 16884231 in provider escience_a40 proc_lsst.multi INFO: found job 16884231 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:41 proc_lsst.multi:146 [INFO] found job 16884232 in provider checkpoint proc_lsst.multi INFO: found job 16884232 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884232,16884233,16884234,16884235,16884236,16884237' parsl.providers.slurm.slurm DEBUG: squeue returned 16884237 PD 16884232 R 16884233 R 16884234 R 16884235 R 16884236 R parsl.providers.slurm.slurm DEBUG: Updating job 16884237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16884232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884236 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:42 proc_lsst.multi:146 [INFO] found job 16884233 in provider checkpoint proc_lsst.multi INFO: found job 16884233 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884232,16884233,16884234,16884235,16884236,16884237' parsl.providers.slurm.slurm DEBUG: squeue returned 16884237 PD 16884232 R 16884233 R 16884234 R 16884235 R 16884236 R parsl.providers.slurm.slurm DEBUG: Updating job 16884237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16884232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884236 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:42 proc_lsst.multi:146 [INFO] found job 16884234 in provider checkpoint proc_lsst.multi INFO: found job 16884234 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884232,16884233,16884234,16884235,16884236,16884237' parsl.providers.slurm.slurm DEBUG: squeue returned 16884237 PD 16884232 R 16884233 R 16884234 R 16884235 R 16884236 R parsl.providers.slurm.slurm DEBUG: Updating job 16884237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16884232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884236 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:42 proc_lsst.multi:146 [INFO] found job 16884235 in provider checkpoint proc_lsst.multi INFO: found job 16884235 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884232,16884233,16884234,16884235,16884236,16884237' parsl.providers.slurm.slurm DEBUG: squeue returned 16884237 PD 16884232 R 16884233 R 16884234 R 16884235 R 16884236 R parsl.providers.slurm.slurm DEBUG: Updating job 16884237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16884232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884236 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:42 proc_lsst.multi:146 [INFO] found job 16884236 in provider checkpoint proc_lsst.multi INFO: found job 16884236 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884232,16884233,16884234,16884235,16884236,16884237' parsl.providers.slurm.slurm DEBUG: squeue returned 16884237 PD 16884232 R 16884233 R 16884234 R 16884235 R 16884236 R parsl.providers.slurm.slurm DEBUG: Updating job 16884237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16884232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884236 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:35:42 proc_lsst.multi:146 [INFO] found job 16884237 in provider checkpoint proc_lsst.multi INFO: found job 16884237 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884232,16884233,16884234,16884235,16884236,16884237' parsl.providers.slurm.slurm DEBUG: squeue returned 16884237 PD 16884232 R 16884233 R 16884234 R 16884235 R 16884236 R parsl.providers.slurm.slurm DEBUG: Updating job 16884237 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16884232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884236 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 28 parsl.dataflow.strategy DEBUG: Executor multi has 28 active tasks, 61/1 running/pending blocks, and 97 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22988724935824 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 28 parsl.dataflow.strategy DEBUG: Executor multi has 28 active tasks, 61/1 running/pending blocks, and 97 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22988724935824 parsl.dataflow.dflow INFO: Task 49 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7dfc33d7d1b93df3648c565403cb04e3 with result from task 49 parsl.dataflow.dflow DEBUG: Task 49 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/275d9377-c990-4aba-9d0f-4c5e3c686273_isrForCrosstalkSources_1123954_47.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/275d9377-c990-4aba-9d0f-4c5e3c686273_isrForCrosstalkSources_1123954_47.stderr parsl.dataflow.dflow INFO: Task 41 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38319fc7aad97bff1601e9592978ceb5 with result from task 41 parsl.dataflow.dflow DEBUG: Task 41 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/872f7a5b-4023-47d5-a21c-a9953b878572_isrForCrosstalkSources_1123954_48.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/872f7a5b-4023-47d5-a21c-a9953b878572_isrForCrosstalkSources_1123954_48.stderr parsl.dataflow.dflow INFO: Task 47 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32ad066e1da9b954bc67a0d0f3c9326d with result from task 47 parsl.dataflow.dflow DEBUG: Task 47 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/0fbc9a99-03c9-44fd-8a86-7f4a67630c5b_isrForCrosstalkSources_1123958_17.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/0fbc9a99-03c9-44fd-8a86-7f4a67630c5b_isrForCrosstalkSources_1123958_17.stderr parsl.dataflow.dflow INFO: Task 39 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c33000ac77f1b27d48a4e9b163ce9ef with result from task 39 parsl.dataflow.dflow DEBUG: Task 39 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/0b2216f5-93a9-404f-83af-960f718e0bbf_isrForCrosstalkSources_1123956_11.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/0b2216f5-93a9-404f-83af-960f718e0bbf_isrForCrosstalkSources_1123956_11.stderr parsl.dataflow.dflow INFO: Task 50 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry adf2795b707fe4f0c8149c416a31ea1c with result from task 50 parsl.dataflow.dflow DEBUG: Task 50 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/4f4208b7-fc72-4cc3-ba61-f5735ff24222_isrForCrosstalkSources_1123958_26.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/4f4208b7-fc72-4cc3-ba61-f5735ff24222_isrForCrosstalkSources_1123958_26.stderr parsl.dataflow.dflow INFO: Task 43 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2178bb7707aaff08e2e029bd3f3f0197 with result from task 43 parsl.dataflow.dflow DEBUG: Task 43 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/20912cf1-26de-4a61-b947-603f7f179329_isrForCrosstalkSources_1123956_19.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/20912cf1-26de-4a61-b947-603f7f179329_isrForCrosstalkSources_1123956_19.stderr parsl.dataflow.dflow INFO: Task 48 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c8ec3a8094e62d1fc10d380838e8548 with result from task 48 parsl.dataflow.dflow DEBUG: Task 48 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/0cc450a1-4c04-4953-9b72-f1b6e2a5cfae_isrForCrosstalkSources_1123956_7.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/0cc450a1-4c04-4953-9b72-f1b6e2a5cfae_isrForCrosstalkSources_1123956_7.stderr parsl.dataflow.dflow INFO: Task 42 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 012d356e22ceca5b10d768eeefea7265 with result from task 42 parsl.dataflow.dflow DEBUG: Task 42 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/0ce32f18-e3a4-443b-a02d-94a9aa1c345f_isrForCrosstalkSources_1123956_21.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/0ce32f18-e3a4-443b-a02d-94a9aa1c345f_isrForCrosstalkSources_1123956_21.stderr parsl.dataflow.dflow INFO: Task 46 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b4e4af1d77c797fe9f0c965de258100 with result from task 46 parsl.dataflow.dflow DEBUG: Task 46 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/506bba7e-88df-4fb3-bdfc-4abb29669499_isrForCrosstalkSources_1123956_15.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/506bba7e-88df-4fb3-bdfc-4abb29669499_isrForCrosstalkSources_1123956_15.stderr parsl.dataflow.dflow INFO: Task 45 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3801b8b790fbfe980663ba56b4fab2b0 with result from task 45 parsl.dataflow.dflow DEBUG: Task 45 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/95e69fd7-6fe5-4de6-9fae-acabd1eb86ed_isrForCrosstalkSources_1124087_54.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/95e69fd7-6fe5-4de6-9fae-acabd1eb86ed_isrForCrosstalkSources_1124087_54.stderr parsl.dataflow.dflow INFO: Task 44 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 310dd9e185726bf53f9e8cc0fcc52af4 with result from task 44 parsl.dataflow.dflow DEBUG: Task 44 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/cee6d43b-65ae-40a9-9743-5db753a02cc6_isrForCrosstalkSources_1123954_27.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/cee6d43b-65ae-40a9-9743-5db753a02cc6_isrForCrosstalkSources_1123954_27.stderr parsl.dataflow.dflow INFO: Task 38 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a725c27d446ff7837cae0f25c52a83d8 with result from task 38 parsl.dataflow.dflow DEBUG: Task 38 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/c1774e1a-3227-4b20-b606-018197913b07_isrForCrosstalkSources_1124087_33.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/c1774e1a-3227-4b20-b606-018197913b07_isrForCrosstalkSources_1124087_33.stderr parsl.dataflow.dflow INFO: Task 40 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 067ac8eeb962dd30ecc75fb8327a3e66 with result from task 40 parsl.dataflow.dflow DEBUG: Task 40 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/eb75578c-d5de-4eba-a593-41aa7b72882d_isrForCrosstalkSources_1123956_29.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/eb75578c-d5de-4eba-a593-41aa7b72882d_isrForCrosstalkSources_1123956_29.stderr parsl.dataflow.dflow INFO: Task 55 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d5993f0e746c523fd74a2a78cbc081c with result from task 55 parsl.dataflow.dflow DEBUG: Task 55 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/aee6b67b-c26d-42a8-bef5-4c24e043e4ec_isrForCrosstalkSources_1123954_43.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/aee6b67b-c26d-42a8-bef5-4c24e043e4ec_isrForCrosstalkSources_1123954_43.stderr parsl.dataflow.dflow INFO: Task 51 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d144575f2ad57d23e1a5bdc688a3248c with result from task 51 parsl.dataflow.dflow DEBUG: Task 51 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/f173d899-4afa-40f8-8afd-1e36a4784455_isrForCrosstalkSources_1123958_50.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/f173d899-4afa-40f8-8afd-1e36a4784455_isrForCrosstalkSources_1123958_50.stderr parsl.dataflow.dflow INFO: Task 54 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c4cd2becef2b60a85e8f235cfd15680 with result from task 54 parsl.dataflow.dflow DEBUG: Task 54 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/bc1d1bf8-d843-40ef-bf68-135ca854f8ab_isrForCrosstalkSources_1123958_1.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/bc1d1bf8-d843-40ef-bf68-135ca854f8ab_isrForCrosstalkSources_1123958_1.stderr parsl.dataflow.dflow INFO: Task 57 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b370d1356913ca021fa0237f5909fca with result from task 57 parsl.dataflow.dflow DEBUG: Task 57 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/0d1a2d9f-e8e9-4684-bee5-05bb484cda3d_isrForCrosstalkSources_1123954_14.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/0d1a2d9f-e8e9-4684-bee5-05bb484cda3d_isrForCrosstalkSources_1123954_14.stderr parsl.dataflow.dflow INFO: Task 56 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5f5f2bef0b0efa432a96f605d41642b with result from task 56 parsl.dataflow.dflow DEBUG: Task 56 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/9fac5b87-9c0a-46c3-a2c4-669ae8b9c81f_isrForCrosstalkSources_1123954_3.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/9fac5b87-9c0a-46c3-a2c4-669ae8b9c81f_isrForCrosstalkSources_1123954_3.stderr parsl.dataflow.dflow INFO: Task 52 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec360ba692371a2e80cfd1f8d949c526 with result from task 52 parsl.dataflow.dflow DEBUG: Task 52 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/576bbdae-d433-4f36-bad3-e66a577e288b_isrForCrosstalkSources_1124087_6.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/576bbdae-d433-4f36-bad3-e66a577e288b_isrForCrosstalkSources_1124087_6.stderr parsl.dataflow.dflow INFO: Task 53 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b6403c65aacc6a75007b7f361da5de2 with result from task 53 parsl.dataflow.dflow DEBUG: Task 53 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/1b50ad35-57fc-4140-a7bc-35209ba4f44b_isrForCrosstalkSources_1124087_44.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/1b50ad35-57fc-4140-a7bc-35209ba4f44b_isrForCrosstalkSources_1124087_44.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 8 parsl.dataflow.strategy DEBUG: Executor multi has 8 active tasks, 61/1 running/pending blocks, and 97 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22988724935824 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 8 parsl.dataflow.strategy DEBUG: Executor multi has 8 active tasks, 61/1 running/pending blocks, and 97 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22988724935824 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 8 parsl.dataflow.strategy DEBUG: Executor multi has 8 active tasks, 61/1 running/pending blocks, and 97 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22988724935824 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 8 parsl.dataflow.strategy DEBUG: Executor multi has 8 active tasks, 61/1 running/pending blocks, and 97 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22988724935824 2024-03-07 03:36:07 proc_lsst.multi:146 [INFO] found job 766 in provider local proc_lsst.multi INFO: found job 766 in provider local 2024-03-07 03:36:07 proc_lsst.multi:146 [INFO] found job 16884172 in provider astro proc_lsst.multi INFO: found job 16884172 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:08 proc_lsst.multi:146 [INFO] found job 16884173 in provider astro proc_lsst.multi INFO: found job 16884173 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:08 proc_lsst.multi:146 [INFO] found job 16884174 in provider astro proc_lsst.multi INFO: found job 16884174 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:08 proc_lsst.multi:146 [INFO] found job 16884175 in provider astro proc_lsst.multi INFO: found job 16884175 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:08 proc_lsst.multi:146 [INFO] found job 16884176 in provider astro proc_lsst.multi INFO: found job 16884176 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.dataflow.dflow INFO: Task 72 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 645b0960a80e89c5f8cac409179c0b6b with result from task 72 parsl.dataflow.dflow DEBUG: Task 72 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/42543483-f2d0-46f1-bc7f-09b0006f1ef0_isrForCrosstalkSources_1123958_48.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123958/42543483-f2d0-46f1-bc7f-09b0006f1ef0_isrForCrosstalkSources_1123958_48.stderr parsl.dataflow.dflow INFO: Task 74 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c187c9a79cb2f99d3c22a13c9fc1b3a0 with result from task 74 parsl.dataflow.dflow DEBUG: Task 74 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/aa83b164-f09d-404c-ace9-43d5fc9f6001_isrForCrosstalkSources_1123956_37.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/aa83b164-f09d-404c-ace9-43d5fc9f6001_isrForCrosstalkSources_1123956_37.stderr parsl.dataflow.dflow INFO: Task 70 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2be97232539d66c9029570f44f769428 with result from task 70 parsl.dataflow.dflow DEBUG: Task 70 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/291d2d0e-ae5b-425e-95f4-acd7833cad8b_isrForCrosstalkSources_1123954_34.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/291d2d0e-ae5b-425e-95f4-acd7833cad8b_isrForCrosstalkSources_1123954_34.stderr parsl.dataflow.dflow INFO: Task 69 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da6614574f32209a8b9e158a3d08e88c with result from task 69 parsl.dataflow.dflow DEBUG: Task 69 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/76ac6ef4-2f69-446d-bde8-fb6b9ef838fb_isrForCrosstalkSources_1123956_54.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123956/76ac6ef4-2f69-446d-bde8-fb6b9ef838fb_isrForCrosstalkSources_1123956_54.stderr parsl.dataflow.dflow INFO: Task 73 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 811d5c9297d1ef2daa1e66504f708ac3 with result from task 73 parsl.dataflow.dflow DEBUG: Task 73 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/06000f64-d9f4-4b27-b1d9-eb2b79373752_isrForCrosstalkSources_1123954_24.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/06000f64-d9f4-4b27-b1d9-eb2b79373752_isrForCrosstalkSources_1123954_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:08 proc_lsst.multi:146 [INFO] found job 16884177 in provider astro proc_lsst.multi INFO: found job 16884177 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:08 proc_lsst.multi:146 [INFO] found job 16884178 in provider astro proc_lsst.multi INFO: found job 16884178 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.dataflow.dflow INFO: Task 67 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df642638f72b36d57b87ad1a321ede9d with result from task 67 parsl.dataflow.dflow DEBUG: Task 67 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/b5794b80-9059-4385-977e-aec0d132e02c_isrForCrosstalkSources_1123954_39.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1123954/b5794b80-9059-4385-977e-aec0d132e02c_isrForCrosstalkSources_1123954_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:08 proc_lsst.multi:146 [INFO] found job 16884179 in provider astro proc_lsst.multi INFO: found job 16884179 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:08 proc_lsst.multi:146 [INFO] found job 16884180 in provider astro proc_lsst.multi INFO: found job 16884180 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:09 proc_lsst.multi:146 [INFO] found job 16884181 in provider astro proc_lsst.multi INFO: found job 16884181 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:09 proc_lsst.multi:146 [INFO] found job 16884182 in provider astro proc_lsst.multi INFO: found job 16884182 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:09 proc_lsst.multi:146 [INFO] found job 16884183 in provider astro proc_lsst.multi INFO: found job 16884183 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.dataflow.dflow INFO: Task 71 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d351488a0cddacd1f5de9027417f4404 with result from task 71 parsl.dataflow.dflow DEBUG: Task 71 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/fbc4616f-1322-46f5-914a-a140b66a14a1_isrForCrosstalkSources_1124087_28.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/fbc4616f-1322-46f5-914a-a140b66a14a1_isrForCrosstalkSources_1124087_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:09 proc_lsst.multi:146 [INFO] found job 16884184 in provider astro proc_lsst.multi INFO: found job 16884184 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:09 proc_lsst.multi:146 [INFO] found job 16884185 in provider astro proc_lsst.multi INFO: found job 16884185 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:09 proc_lsst.multi:146 [INFO] found job 16884186 in provider astro proc_lsst.multi INFO: found job 16884186 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:09 proc_lsst.multi:146 [INFO] found job 16884187 in provider astro proc_lsst.multi INFO: found job 16884187 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:09 proc_lsst.multi:146 [INFO] found job 16884188 in provider astro proc_lsst.multi INFO: found job 16884188 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:10 proc_lsst.multi:146 [INFO] found job 16884189 in provider astro proc_lsst.multi INFO: found job 16884189 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:10 proc_lsst.multi:146 [INFO] found job 16884190 in provider astro proc_lsst.multi INFO: found job 16884190 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:10 proc_lsst.multi:146 [INFO] found job 16884191 in provider astro proc_lsst.multi INFO: found job 16884191 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:10 proc_lsst.multi:146 [INFO] found job 16884192 in provider astro proc_lsst.multi INFO: found job 16884192 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:10 proc_lsst.multi:146 [INFO] found job 16884193 in provider astro proc_lsst.multi INFO: found job 16884193 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:10 proc_lsst.multi:146 [INFO] found job 16884194 in provider astro proc_lsst.multi INFO: found job 16884194 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:10 proc_lsst.multi:146 [INFO] found job 16884195 in provider astro proc_lsst.multi INFO: found job 16884195 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:11 proc_lsst.multi:146 [INFO] found job 16884196 in provider astro proc_lsst.multi INFO: found job 16884196 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:11 proc_lsst.multi:146 [INFO] found job 16884197 in provider astro proc_lsst.multi INFO: found job 16884197 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:11 proc_lsst.multi:146 [INFO] found job 16884203 in provider astro proc_lsst.multi INFO: found job 16884203 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:11 proc_lsst.multi:146 [INFO] found job 16884204 in provider astro proc_lsst.multi INFO: found job 16884204 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:11 proc_lsst.multi:146 [INFO] found job 16884205 in provider astro proc_lsst.multi INFO: found job 16884205 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:11 proc_lsst.multi:146 [INFO] found job 16884206 in provider astro proc_lsst.multi INFO: found job 16884206 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884172,16884173,16884174,16884175,16884176,16884177,16884178,16884179,16884180,16884181,16884182,16884183,16884184,16884185,16884186,16884187,16884188,16884189,16884190,16884191,16884192,16884193,16884194,16884195,16884196,16884197,16884203,16884204,16884205,16884206' parsl.providers.slurm.slurm DEBUG: squeue returned 16884206 R 16884204 R 16884205 R 16884203 R 16884183 R 16884184 R 16884185 R 16884186 R 16884187 R 16884188 R 16884189 R 16884190 R 16884191 R 16884192 R 16884193 R 16884194 R 16884195 R 16884196 R 16884197 R 16884173 R 16884174 R 16884175 R 16884176 R 16884177 R 16884178 R 16884179 R 16884180 R 16884181 R 16884182 R 16884172 R parsl.providers.slurm.slurm DEBUG: Updating job 16884206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884196 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884172 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:11 proc_lsst.multi:146 [INFO] found job 16884207 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884207 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884207,16884208,16884209,16884210,16884211' parsl.providers.slurm.slurm DEBUG: squeue returned 16884207 R 16884208 R 16884209 R 16884210 R 16884211 R parsl.providers.slurm.slurm DEBUG: Updating job 16884207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884211 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:11 proc_lsst.multi:146 [INFO] found job 16884208 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884208 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884207,16884208,16884209,16884210,16884211' parsl.providers.slurm.slurm DEBUG: squeue returned 16884207 R 16884208 R 16884209 R 16884210 R 16884211 R parsl.providers.slurm.slurm DEBUG: Updating job 16884207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884211 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:12 proc_lsst.multi:146 [INFO] found job 16884209 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884209 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884207,16884208,16884209,16884210,16884211' parsl.providers.slurm.slurm DEBUG: squeue returned 16884207 R 16884208 R 16884209 R 16884210 R 16884211 R parsl.providers.slurm.slurm DEBUG: Updating job 16884207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884211 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:12 proc_lsst.multi:146 [INFO] found job 16884210 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884210 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884207,16884208,16884209,16884210,16884211' parsl.providers.slurm.slurm DEBUG: squeue returned 16884207 R 16884208 R 16884209 R 16884210 R 16884211 R parsl.providers.slurm.slurm DEBUG: Updating job 16884207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884211 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:12 proc_lsst.multi:146 [INFO] found job 16884211 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884211 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884207,16884208,16884209,16884210,16884211' parsl.providers.slurm.slurm DEBUG: squeue returned 16884207 R 16884208 R 16884209 R 16884210 R 16884211 R parsl.providers.slurm.slurm DEBUG: Updating job 16884207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884211 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:12 proc_lsst.multi:146 [INFO] found job 16884212 in provider escience_a40 proc_lsst.multi INFO: found job 16884212 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:12 proc_lsst.multi:146 [INFO] found job 16884213 in provider escience_a40 proc_lsst.multi INFO: found job 16884213 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:12 proc_lsst.multi:146 [INFO] found job 16884214 in provider escience_a40 proc_lsst.multi INFO: found job 16884214 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:12 proc_lsst.multi:146 [INFO] found job 16884215 in provider escience_a40 proc_lsst.multi INFO: found job 16884215 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:12 proc_lsst.multi:146 [INFO] found job 16884216 in provider escience_a40 proc_lsst.multi INFO: found job 16884216 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:13 proc_lsst.multi:146 [INFO] found job 16884217 in provider escience_a40 proc_lsst.multi INFO: found job 16884217 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:13 proc_lsst.multi:146 [INFO] found job 16884218 in provider escience_a40 proc_lsst.multi INFO: found job 16884218 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:13 proc_lsst.multi:146 [INFO] found job 16884219 in provider escience_a40 proc_lsst.multi INFO: found job 16884219 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:13 proc_lsst.multi:146 [INFO] found job 16884220 in provider escience_a40 proc_lsst.multi INFO: found job 16884220 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:13 proc_lsst.multi:146 [INFO] found job 16884221 in provider escience_a40 proc_lsst.multi INFO: found job 16884221 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:13 proc_lsst.multi:146 [INFO] found job 16884222 in provider escience_a40 proc_lsst.multi INFO: found job 16884222 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:13 proc_lsst.multi:146 [INFO] found job 16884223 in provider escience_a40 proc_lsst.multi INFO: found job 16884223 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:13 proc_lsst.multi:146 [INFO] found job 16884224 in provider escience_a40 proc_lsst.multi INFO: found job 16884224 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:14 proc_lsst.multi:146 [INFO] found job 16884225 in provider escience_a40 proc_lsst.multi INFO: found job 16884225 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:14 proc_lsst.multi:146 [INFO] found job 16884226 in provider escience_a40 proc_lsst.multi INFO: found job 16884226 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:14 proc_lsst.multi:146 [INFO] found job 16884227 in provider escience_a40 proc_lsst.multi INFO: found job 16884227 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:14 proc_lsst.multi:146 [INFO] found job 16884228 in provider escience_a40 proc_lsst.multi INFO: found job 16884228 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:14 proc_lsst.multi:146 [INFO] found job 16884229 in provider escience_a40 proc_lsst.multi INFO: found job 16884229 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:14 proc_lsst.multi:146 [INFO] found job 16884230 in provider escience_a40 proc_lsst.multi INFO: found job 16884230 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:14 proc_lsst.multi:146 [INFO] found job 16884231 in provider escience_a40 proc_lsst.multi INFO: found job 16884231 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884212,16884213,16884214,16884215,16884216,16884217,16884218,16884219,16884220,16884221,16884222,16884223,16884224,16884225,16884226,16884227,16884228,16884229,16884230,16884231' parsl.providers.slurm.slurm DEBUG: squeue returned 16884223 R 16884224 R 16884225 R 16884226 R 16884227 R 16884228 R 16884229 R 16884230 R 16884231 R 16884217 R 16884218 R 16884219 R 16884220 R 16884221 R 16884222 R 16884212 R 16884213 R 16884214 R 16884215 R 16884216 R parsl.providers.slurm.slurm DEBUG: Updating job 16884223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884227 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884212 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884216 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:16 proc_lsst.multi:146 [INFO] found job 16884232 in provider checkpoint proc_lsst.multi INFO: found job 16884232 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884232,16884233,16884234,16884235,16884236,16884237' parsl.providers.slurm.slurm DEBUG: squeue returned 16884237 R 16884232 R 16884233 R 16884234 R 16884235 R 16884236 R parsl.providers.slurm.slurm DEBUG: Updating job 16884237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884236 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:17 proc_lsst.multi:146 [INFO] found job 16884233 in provider checkpoint proc_lsst.multi INFO: found job 16884233 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884232,16884233,16884234,16884235,16884236,16884237' parsl.providers.slurm.slurm DEBUG: squeue returned 16884237 R 16884232 R 16884233 R 16884234 R 16884235 R 16884236 R parsl.providers.slurm.slurm DEBUG: Updating job 16884237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884236 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:17 proc_lsst.multi:146 [INFO] found job 16884234 in provider checkpoint proc_lsst.multi INFO: found job 16884234 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884232,16884233,16884234,16884235,16884236,16884237' parsl.dataflow.dflow INFO: Task 68 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af61fac49737e1910d04fd2b20a08e5b with result from task 68 parsl.dataflow.dflow INFO: DFK cleanup initiated parsl.dataflow.dflow INFO: Summary of tasks in DFK: parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0 parsl.dataflow.dflow INFO: Tasks in state States.pending: 0 parsl.dataflow.dflow INFO: Tasks in state States.running: 0 parsl.dataflow.dflow DEBUG: Task 68 checkpointed parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 248 parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Tasks in state States.failed: 0 parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/937a37e1-cd9e-4458-b333-4aea36e6ed9d_isrForCrosstalkSources_1124087_62.stdout parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 0 parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/submit/DEEP/20220821/B1m/science#step0/20240307T033310Z/logs/isrForCrosstalkSources/1124087/937a37e1-cd9e-4458-b333-4aea36e6ed9d_isrForCrosstalkSources_1124087_62.stderr parsl.dataflow.dflow INFO: Tasks in state States.launched: 0 parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0 parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0 parsl.dataflow.dflow INFO: Tasks in state States.joining: 0 parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0 parsl.dataflow.dflow INFO: End of summary parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Closing job status poller parsl.providers.slurm.slurm DEBUG: squeue returned 16884237 R 16884232 R 16884233 R 16884234 R 16884235 R 16884236 R parsl.providers.slurm.slurm DEBUG: Updating job 16884237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884236 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:17 proc_lsst.multi:146 [INFO] found job 16884235 in provider checkpoint proc_lsst.multi INFO: found job 16884235 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884232,16884233,16884234,16884235,16884236,16884237' parsl.providers.slurm.slurm DEBUG: squeue returned 16884237 R 16884232 R 16884233 R 16884234 R 16884235 R 16884236 R parsl.providers.slurm.slurm DEBUG: Updating job 16884237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884236 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:17 proc_lsst.multi:146 [INFO] found job 16884236 in provider checkpoint proc_lsst.multi INFO: found job 16884236 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884232,16884233,16884234,16884235,16884236,16884237' parsl.providers.slurm.slurm DEBUG: squeue returned 16884237 R 16884232 R 16884233 R 16884234 R 16884235 R 16884236 R parsl.providers.slurm.slurm DEBUG: Updating job 16884237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884236 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:36:17 proc_lsst.multi:146 [INFO] found job 16884237 in provider checkpoint proc_lsst.multi INFO: found job 16884237 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884232,16884233,16884234,16884235,16884236,16884237' parsl.providers.slurm.slurm DEBUG: squeue returned 16884237 R 16884232 R 16884233 R 16884234 R 16884235 R 16884236 R parsl.providers.slurm.slurm DEBUG: Updating job 16884237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884236 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 0 parsl.dataflow.strategy DEBUG: Executor multi has 0 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 1: Executor has no active tasks parsl.dataflow.strategy DEBUG: Strategy case 1b: Executor has no active tasks, and more (62) than minimum blocks (0) parsl.dataflow.strategy DEBUG: Starting idle timer for executor. If idle time exceeds 120.0s, blocks will be scaled in parsl.dataflow.strategy DEBUG: Idle time 2.6226043701171875e-06s is less than max_idletime 120.0s for executor multi; not scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22988724935824 parsl.dataflow.dflow INFO: Terminated job status poller parsl.dataflow.dflow INFO: Scaling in and shutting down executors parsl.dataflow.dflow INFO: Scaling in executor multi parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=62, block_ids=[] parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 62 blocks parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a61dba3f9022 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a61dba3f9022 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ac3cca1d436f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ac3cca1d436f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5dc95113465a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5dc95113465a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fb2e804f6148 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fb2e804f6148 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e577fd26e8d5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e577fd26e8d5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 77510847879e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 77510847879e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1cf929df7c0e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1cf929df7c0e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 361a89362d37 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 361a89362d37 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5136a524988e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5136a524988e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e498b3a69db4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e498b3a69db4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3161d538dbd9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3161d538dbd9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1995bf4fd582 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1995bf4fd582 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 51d6f70fbd65 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 51d6f70fbd65 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e4a4f4aba2b7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e4a4f4aba2b7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2a837bbc5ec1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2a837bbc5ec1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c3be5af48cbc parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c3be5af48cbc parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 147f53cb8ba6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 147f53cb8ba6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 004e40083b3e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 004e40083b3e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 042d1eb5819b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 042d1eb5819b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 66a75db612f2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 66a75db612f2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8dc1a9f6e4b1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8dc1a9f6e4b1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4fb8d3662e23 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4fb8d3662e23 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1ce8036870a2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1ce8036870a2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b1b6b5020a40 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b1b6b5020a40 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 26db783d2a73 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 26db783d2a73 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e9c049108b62 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e9c049108b62 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fbdf1c26e7f8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fbdf1c26e7f8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b101ecddc0bd parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b101ecddc0bd parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: abdf9369c065 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: abdf9369c065 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f751e5580365 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f751e5580365 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 39d04ce8bfcf parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 39d04ce8bfcf parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6690714df4da parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6690714df4da parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a6bf7ae87aea parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a6bf7ae87aea parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9757746911ed parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9757746911ed parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5eb3afe25305 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5eb3afe25305 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dac34c082ac5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dac34c082ac5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8d8a1c1b89a8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8d8a1c1b89a8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b69898ff2aab parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b69898ff2aab parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 025e93378b7f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 025e93378b7f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 752843b1cb1e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 752843b1cb1e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0949e79a7cb0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0949e79a7cb0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ace8f40a96d4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ace8f40a96d4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2206993e16b7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2206993e16b7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 184aaf51bba9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 184aaf51bba9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1cfc5b2b8ab1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1cfc5b2b8ab1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ff5821131ea0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ff5821131ea0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 829b776c6c14 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 829b776c6c14 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8011890e5fc3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8011890e5fc3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 254418b2af60 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 254418b2af60 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b11f8186e2aa parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b11f8186e2aa parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0aaa4022ec72 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0aaa4022ec72 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 369439a23e5f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 369439a23e5f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5525eb51f7b0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5525eb51f7b0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 13a450a992d4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 13a450a992d4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 490f2ede8036 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 490f2ede8036 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 578903a05570 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 578903a05570 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9180893b8ee0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9180893b8ee0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5bf6fb733e98 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5bf6fb733e98 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 145d24fc6ccd parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 145d24fc6ccd parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fc716fbd1f7a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fc716fbd1f7a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 78c5c7921add parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 78c5c7921add parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 07c4de408567 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 07c4de408567 2024-03-07 03:36:17 proc_lsst.multi:146 [INFO] found job 16884234 in provider checkpoint proc_lsst.multi INFO: found job 16884234 in provider checkpoint 2024-03-07 03:36:17 proc_lsst.multi:201 [INFO] cancelling 16884234 on provider checkpoint proc_lsst.multi INFO: cancelling 16884234 on provider checkpoint 2024-03-07 03:36:17 proc_lsst.multi:146 [INFO] found job 16884237 in provider checkpoint proc_lsst.multi INFO: found job 16884237 in provider checkpoint 2024-03-07 03:36:17 proc_lsst.multi:201 [INFO] cancelling 16884237 on provider checkpoint proc_lsst.multi INFO: cancelling 16884237 on provider checkpoint 2024-03-07 03:36:18 proc_lsst.multi:146 [INFO] found job 16884214 in provider escience_a40 proc_lsst.multi INFO: found job 16884214 in provider escience_a40 2024-03-07 03:36:18 proc_lsst.multi:201 [INFO] cancelling 16884214 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884214 on provider escience_a40 2024-03-07 03:36:18 proc_lsst.multi:146 [INFO] found job 16884212 in provider escience_a40 proc_lsst.multi INFO: found job 16884212 in provider escience_a40 2024-03-07 03:36:18 proc_lsst.multi:201 [INFO] cancelling 16884212 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884212 on provider escience_a40 2024-03-07 03:36:18 proc_lsst.multi:146 [INFO] found job 16884223 in provider escience_a40 proc_lsst.multi INFO: found job 16884223 in provider escience_a40 2024-03-07 03:36:18 proc_lsst.multi:201 [INFO] cancelling 16884223 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884223 on provider escience_a40 2024-03-07 03:36:18 proc_lsst.multi:146 [INFO] found job 16884215 in provider escience_a40 proc_lsst.multi INFO: found job 16884215 in provider escience_a40 2024-03-07 03:36:18 proc_lsst.multi:201 [INFO] cancelling 16884215 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884215 on provider escience_a40 2024-03-07 03:36:18 proc_lsst.multi:146 [INFO] found job 16884216 in provider escience_a40 proc_lsst.multi INFO: found job 16884216 in provider escience_a40 2024-03-07 03:36:18 proc_lsst.multi:201 [INFO] cancelling 16884216 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884216 on provider escience_a40 2024-03-07 03:36:18 proc_lsst.multi:146 [INFO] found job 16884217 in provider escience_a40 proc_lsst.multi INFO: found job 16884217 in provider escience_a40 2024-03-07 03:36:18 proc_lsst.multi:201 [INFO] cancelling 16884217 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884217 on provider escience_a40 2024-03-07 03:36:18 proc_lsst.multi:146 [INFO] found job 16884221 in provider escience_a40 proc_lsst.multi INFO: found job 16884221 in provider escience_a40 2024-03-07 03:36:18 proc_lsst.multi:201 [INFO] cancelling 16884221 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884221 on provider escience_a40 2024-03-07 03:36:18 proc_lsst.multi:146 [INFO] found job 16884222 in provider escience_a40 proc_lsst.multi INFO: found job 16884222 in provider escience_a40 2024-03-07 03:36:18 proc_lsst.multi:201 [INFO] cancelling 16884222 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884222 on provider escience_a40 2024-03-07 03:36:18 proc_lsst.multi:146 [INFO] found job 16884231 in provider escience_a40 proc_lsst.multi INFO: found job 16884231 in provider escience_a40 2024-03-07 03:36:18 proc_lsst.multi:201 [INFO] cancelling 16884231 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884231 on provider escience_a40 2024-03-07 03:36:18 proc_lsst.multi:146 [INFO] found job 16884227 in provider escience_a40 proc_lsst.multi INFO: found job 16884227 in provider escience_a40 2024-03-07 03:36:18 proc_lsst.multi:201 [INFO] cancelling 16884227 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884227 on provider escience_a40 2024-03-07 03:36:18 proc_lsst.multi:146 [INFO] found job 16884226 in provider escience_a40 proc_lsst.multi INFO: found job 16884226 in provider escience_a40 2024-03-07 03:36:18 proc_lsst.multi:201 [INFO] cancelling 16884226 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884226 on provider escience_a40 2024-03-07 03:36:19 proc_lsst.multi:146 [INFO] found job 16884213 in provider escience_a40 proc_lsst.multi INFO: found job 16884213 in provider escience_a40 2024-03-07 03:36:19 proc_lsst.multi:201 [INFO] cancelling 16884213 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884213 on provider escience_a40 2024-03-07 03:36:19 proc_lsst.multi:146 [INFO] found job 16884220 in provider escience_a40 proc_lsst.multi INFO: found job 16884220 in provider escience_a40 2024-03-07 03:36:19 proc_lsst.multi:201 [INFO] cancelling 16884220 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884220 on provider escience_a40 2024-03-07 03:36:19 proc_lsst.multi:146 [INFO] found job 16884219 in provider escience_a40 proc_lsst.multi INFO: found job 16884219 in provider escience_a40 2024-03-07 03:36:19 proc_lsst.multi:201 [INFO] cancelling 16884219 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884219 on provider escience_a40 2024-03-07 03:36:19 proc_lsst.multi:146 [INFO] found job 16884218 in provider escience_a40 proc_lsst.multi INFO: found job 16884218 in provider escience_a40 2024-03-07 03:36:19 proc_lsst.multi:201 [INFO] cancelling 16884218 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884218 on provider escience_a40 2024-03-07 03:36:19 proc_lsst.multi:146 [INFO] found job 16884224 in provider escience_a40 proc_lsst.multi INFO: found job 16884224 in provider escience_a40 2024-03-07 03:36:19 proc_lsst.multi:201 [INFO] cancelling 16884224 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884224 on provider escience_a40 2024-03-07 03:36:19 proc_lsst.multi:146 [INFO] found job 16884225 in provider escience_a40 proc_lsst.multi INFO: found job 16884225 in provider escience_a40 2024-03-07 03:36:19 proc_lsst.multi:201 [INFO] cancelling 16884225 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884225 on provider escience_a40 2024-03-07 03:36:19 proc_lsst.multi:146 [INFO] found job 16884229 in provider escience_a40 proc_lsst.multi INFO: found job 16884229 in provider escience_a40 2024-03-07 03:36:19 proc_lsst.multi:201 [INFO] cancelling 16884229 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884229 on provider escience_a40 2024-03-07 03:36:19 proc_lsst.multi:146 [INFO] found job 16884228 in provider escience_a40 proc_lsst.multi INFO: found job 16884228 in provider escience_a40 2024-03-07 03:36:19 proc_lsst.multi:201 [INFO] cancelling 16884228 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884228 on provider escience_a40 2024-03-07 03:36:19 proc_lsst.multi:146 [INFO] found job 16884230 in provider escience_a40 proc_lsst.multi INFO: found job 16884230 in provider escience_a40 2024-03-07 03:36:19 proc_lsst.multi:201 [INFO] cancelling 16884230 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884230 on provider escience_a40 2024-03-07 03:36:19 proc_lsst.multi:146 [INFO] found job 16884232 in provider checkpoint proc_lsst.multi INFO: found job 16884232 in provider checkpoint 2024-03-07 03:36:19 proc_lsst.multi:201 [INFO] cancelling 16884232 on provider checkpoint proc_lsst.multi INFO: cancelling 16884232 on provider checkpoint 2024-03-07 03:36:19 proc_lsst.multi:146 [INFO] found job 16884187 in provider astro proc_lsst.multi INFO: found job 16884187 in provider astro 2024-03-07 03:36:19 proc_lsst.multi:201 [INFO] cancelling 16884187 on provider astro proc_lsst.multi INFO: cancelling 16884187 on provider astro 2024-03-07 03:36:20 proc_lsst.multi:146 [INFO] found job 16884185 in provider astro proc_lsst.multi INFO: found job 16884185 in provider astro 2024-03-07 03:36:20 proc_lsst.multi:201 [INFO] cancelling 16884185 on provider astro proc_lsst.multi INFO: cancelling 16884185 on provider astro 2024-03-07 03:36:20 proc_lsst.multi:146 [INFO] found job 16884236 in provider checkpoint proc_lsst.multi INFO: found job 16884236 in provider checkpoint 2024-03-07 03:36:20 proc_lsst.multi:201 [INFO] cancelling 16884236 on provider checkpoint proc_lsst.multi INFO: cancelling 16884236 on provider checkpoint 2024-03-07 03:36:20 proc_lsst.multi:146 [INFO] found job 16884195 in provider astro proc_lsst.multi INFO: found job 16884195 in provider astro 2024-03-07 03:36:20 proc_lsst.multi:201 [INFO] cancelling 16884195 on provider astro proc_lsst.multi INFO: cancelling 16884195 on provider astro 2024-03-07 03:36:20 proc_lsst.multi:146 [INFO] found job 16884194 in provider astro proc_lsst.multi INFO: found job 16884194 in provider astro 2024-03-07 03:36:20 proc_lsst.multi:201 [INFO] cancelling 16884194 on provider astro proc_lsst.multi INFO: cancelling 16884194 on provider astro 2024-03-07 03:36:20 proc_lsst.multi:146 [INFO] found job 16884235 in provider checkpoint proc_lsst.multi INFO: found job 16884235 in provider checkpoint 2024-03-07 03:36:20 proc_lsst.multi:201 [INFO] cancelling 16884235 on provider checkpoint proc_lsst.multi INFO: cancelling 16884235 on provider checkpoint 2024-03-07 03:36:20 proc_lsst.multi:146 [INFO] found job 16884177 in provider astro proc_lsst.multi INFO: found job 16884177 in provider astro 2024-03-07 03:36:20 proc_lsst.multi:201 [INFO] cancelling 16884177 on provider astro proc_lsst.multi INFO: cancelling 16884177 on provider astro 2024-03-07 03:36:20 proc_lsst.multi:146 [INFO] found job 16884184 in provider astro proc_lsst.multi INFO: found job 16884184 in provider astro 2024-03-07 03:36:20 proc_lsst.multi:201 [INFO] cancelling 16884184 on provider astro proc_lsst.multi INFO: cancelling 16884184 on provider astro 2024-03-07 03:36:20 proc_lsst.multi:146 [INFO] found job 16884182 in provider astro proc_lsst.multi INFO: found job 16884182 in provider astro 2024-03-07 03:36:20 proc_lsst.multi:201 [INFO] cancelling 16884182 on provider astro proc_lsst.multi INFO: cancelling 16884182 on provider astro 2024-03-07 03:36:20 proc_lsst.multi:146 [INFO] found job 16884204 in provider astro proc_lsst.multi INFO: found job 16884204 in provider astro 2024-03-07 03:36:20 proc_lsst.multi:201 [INFO] cancelling 16884204 on provider astro proc_lsst.multi INFO: cancelling 16884204 on provider astro 2024-03-07 03:36:20 proc_lsst.multi:146 [INFO] found job 16884203 in provider astro proc_lsst.multi INFO: found job 16884203 in provider astro 2024-03-07 03:36:20 proc_lsst.multi:201 [INFO] cancelling 16884203 on provider astro proc_lsst.multi INFO: cancelling 16884203 on provider astro 2024-03-07 03:36:20 proc_lsst.multi:146 [INFO] found job 16884188 in provider astro proc_lsst.multi INFO: found job 16884188 in provider astro 2024-03-07 03:36:20 proc_lsst.multi:201 [INFO] cancelling 16884188 on provider astro proc_lsst.multi INFO: cancelling 16884188 on provider astro 2024-03-07 03:36:21 proc_lsst.multi:146 [INFO] found job 16884186 in provider astro proc_lsst.multi INFO: found job 16884186 in provider astro 2024-03-07 03:36:21 proc_lsst.multi:201 [INFO] cancelling 16884186 on provider astro proc_lsst.multi INFO: cancelling 16884186 on provider astro 2024-03-07 03:36:21 proc_lsst.multi:146 [INFO] found job 16884172 in provider astro proc_lsst.multi INFO: found job 16884172 in provider astro 2024-03-07 03:36:21 proc_lsst.multi:201 [INFO] cancelling 16884172 on provider astro proc_lsst.multi INFO: cancelling 16884172 on provider astro 2024-03-07 03:36:21 proc_lsst.multi:146 [INFO] found job 16884189 in provider astro proc_lsst.multi INFO: found job 16884189 in provider astro 2024-03-07 03:36:21 proc_lsst.multi:201 [INFO] cancelling 16884189 on provider astro proc_lsst.multi INFO: cancelling 16884189 on provider astro 2024-03-07 03:36:21 proc_lsst.multi:146 [INFO] found job 16884173 in provider astro proc_lsst.multi INFO: found job 16884173 in provider astro 2024-03-07 03:36:21 proc_lsst.multi:201 [INFO] cancelling 16884173 on provider astro proc_lsst.multi INFO: cancelling 16884173 on provider astro 2024-03-07 03:36:21 proc_lsst.multi:146 [INFO] found job 16884175 in provider astro proc_lsst.multi INFO: found job 16884175 in provider astro 2024-03-07 03:36:21 proc_lsst.multi:201 [INFO] cancelling 16884175 on provider astro proc_lsst.multi INFO: cancelling 16884175 on provider astro 2024-03-07 03:36:21 proc_lsst.multi:146 [INFO] found job 16884174 in provider astro proc_lsst.multi INFO: found job 16884174 in provider astro 2024-03-07 03:36:21 proc_lsst.multi:201 [INFO] cancelling 16884174 on provider astro proc_lsst.multi INFO: cancelling 16884174 on provider astro 2024-03-07 03:36:21 proc_lsst.multi:146 [INFO] found job 16884180 in provider astro proc_lsst.multi INFO: found job 16884180 in provider astro 2024-03-07 03:36:21 proc_lsst.multi:201 [INFO] cancelling 16884180 on provider astro proc_lsst.multi INFO: cancelling 16884180 on provider astro 2024-03-07 03:36:21 proc_lsst.multi:146 [INFO] found job 16884176 in provider astro proc_lsst.multi INFO: found job 16884176 in provider astro 2024-03-07 03:36:21 proc_lsst.multi:201 [INFO] cancelling 16884176 on provider astro proc_lsst.multi INFO: cancelling 16884176 on provider astro 2024-03-07 03:36:21 proc_lsst.multi:146 [INFO] found job 16884209 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884209 in provider escience_rtx6k 2024-03-07 03:36:21 proc_lsst.multi:201 [INFO] cancelling 16884209 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16884209 on provider escience_rtx6k 2024-03-07 03:36:21 proc_lsst.multi:146 [INFO] found job 16884178 in provider astro proc_lsst.multi INFO: found job 16884178 in provider astro 2024-03-07 03:36:21 proc_lsst.multi:201 [INFO] cancelling 16884178 on provider astro proc_lsst.multi INFO: cancelling 16884178 on provider astro 2024-03-07 03:36:21 proc_lsst.multi:146 [INFO] found job 16884179 in provider astro proc_lsst.multi INFO: found job 16884179 in provider astro 2024-03-07 03:36:21 proc_lsst.multi:201 [INFO] cancelling 16884179 on provider astro proc_lsst.multi INFO: cancelling 16884179 on provider astro 2024-03-07 03:36:21 proc_lsst.multi:146 [INFO] found job 16884210 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884210 in provider escience_rtx6k 2024-03-07 03:36:21 proc_lsst.multi:201 [INFO] cancelling 16884210 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16884210 on provider escience_rtx6k 2024-03-07 03:36:22 proc_lsst.multi:146 [INFO] found job 766 in provider local proc_lsst.multi INFO: found job 766 in provider local 2024-03-07 03:36:22 proc_lsst.multi:201 [INFO] cancelling 766 on provider local proc_lsst.multi INFO: cancelling 766 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 766 2024-03-07 03:36:22 proc_lsst.multi:146 [INFO] found job 16884181 in provider astro proc_lsst.multi INFO: found job 16884181 in provider astro 2024-03-07 03:36:22 proc_lsst.multi:201 [INFO] cancelling 16884181 on provider astro proc_lsst.multi INFO: cancelling 16884181 on provider astro 2024-03-07 03:36:22 proc_lsst.multi:146 [INFO] found job 16884196 in provider astro proc_lsst.multi INFO: found job 16884196 in provider astro 2024-03-07 03:36:22 proc_lsst.multi:201 [INFO] cancelling 16884196 on provider astro proc_lsst.multi INFO: cancelling 16884196 on provider astro 2024-03-07 03:36:22 proc_lsst.multi:146 [INFO] found job 16884197 in provider astro proc_lsst.multi INFO: found job 16884197 in provider astro 2024-03-07 03:36:22 proc_lsst.multi:201 [INFO] cancelling 16884197 on provider astro proc_lsst.multi INFO: cancelling 16884197 on provider astro 2024-03-07 03:36:22 proc_lsst.multi:146 [INFO] found job 16884207 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884207 in provider escience_rtx6k 2024-03-07 03:36:22 proc_lsst.multi:201 [INFO] cancelling 16884207 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16884207 on provider escience_rtx6k 2024-03-07 03:36:22 proc_lsst.multi:146 [INFO] found job 16884208 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884208 in provider escience_rtx6k 2024-03-07 03:36:22 proc_lsst.multi:201 [INFO] cancelling 16884208 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16884208 on provider escience_rtx6k 2024-03-07 03:36:22 proc_lsst.multi:146 [INFO] found job 16884211 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884211 in provider escience_rtx6k 2024-03-07 03:36:22 proc_lsst.multi:201 [INFO] cancelling 16884211 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16884211 on provider escience_rtx6k 2024-03-07 03:36:22 proc_lsst.multi:146 [INFO] found job 16884190 in provider astro proc_lsst.multi INFO: found job 16884190 in provider astro 2024-03-07 03:36:22 proc_lsst.multi:201 [INFO] cancelling 16884190 on provider astro proc_lsst.multi INFO: cancelling 16884190 on provider astro 2024-03-07 03:36:22 proc_lsst.multi:146 [INFO] found job 16884192 in provider astro proc_lsst.multi INFO: found job 16884192 in provider astro 2024-03-07 03:36:22 proc_lsst.multi:201 [INFO] cancelling 16884192 on provider astro proc_lsst.multi INFO: cancelling 16884192 on provider astro 2024-03-07 03:36:23 proc_lsst.multi:146 [INFO] found job 16884206 in provider astro proc_lsst.multi INFO: found job 16884206 in provider astro 2024-03-07 03:36:23 proc_lsst.multi:201 [INFO] cancelling 16884206 on provider astro proc_lsst.multi INFO: cancelling 16884206 on provider astro 2024-03-07 03:36:23 proc_lsst.multi:146 [INFO] found job 16884233 in provider checkpoint proc_lsst.multi INFO: found job 16884233 in provider checkpoint 2024-03-07 03:36:23 proc_lsst.multi:201 [INFO] cancelling 16884233 on provider checkpoint proc_lsst.multi INFO: cancelling 16884233 on provider checkpoint 2024-03-07 03:36:23 proc_lsst.multi:146 [INFO] found job 16884205 in provider astro proc_lsst.multi INFO: found job 16884205 in provider astro 2024-03-07 03:36:23 proc_lsst.multi:201 [INFO] cancelling 16884205 on provider astro proc_lsst.multi INFO: cancelling 16884205 on provider astro 2024-03-07 03:36:23 proc_lsst.multi:146 [INFO] found job 16884191 in provider astro proc_lsst.multi INFO: found job 16884191 in provider astro 2024-03-07 03:36:23 proc_lsst.multi:201 [INFO] cancelling 16884191 on provider astro proc_lsst.multi INFO: cancelling 16884191 on provider astro 2024-03-07 03:36:23 proc_lsst.multi:146 [INFO] found job 16884183 in provider astro proc_lsst.multi INFO: found job 16884183 in provider astro 2024-03-07 03:36:23 proc_lsst.multi:201 [INFO] cancelling 16884183 on provider astro proc_lsst.multi INFO: cancelling 16884183 on provider astro 2024-03-07 03:36:23 proc_lsst.multi:146 [INFO] found job 16884193 in provider astro proc_lsst.multi INFO: found job 16884193 in provider astro 2024-03-07 03:36:23 proc_lsst.multi:201 [INFO] cancelling 16884193 on provider astro proc_lsst.multi INFO: cancelling 16884193 on provider astro parsl.dataflow.dflow INFO: Shutting down executor multi 2024-03-07 03:36:24 proc_lsst.multi:40 [INFO] Cancelling all provider resources proc_lsst.multi INFO: Cancelling all provider resources 2024-03-07 03:36:24 proc_lsst.multi:47 [INFO] new jobs since last cancel ['16884232', '16884233', '16884234', '16884235', '16884236', '16884237', '16884172', '16884173', '16884174', '16884175', '16884176', '16884177', '16884178', '16884179', '16884180', '16884181', '16884182', '16884183', '16884184', '16884185', '16884186', '16884187', '16884188', '16884189', '16884190', '16884191', '16884192', '16884193', '16884194', '16884195', '16884196', '16884197', '16884203', '16884204', '16884205', '16884206', '16884212', '16884213', '16884214', '16884215', '16884216', '16884217', '16884218', '16884219', '16884220', '16884221', '16884222', '16884223', '16884224', '16884225', '16884226', '16884227', '16884228', '16884229', '16884230', '16884231', '16884207', '16884208', '16884209', '16884210', '16884211', '766'] proc_lsst.multi INFO: new jobs since last cancel ['16884232', '16884233', '16884234', '16884235', '16884236', '16884237', '16884172', '16884173', '16884174', '16884175', '16884176', '16884177', '16884178', '16884179', '16884180', '16884181', '16884182', '16884183', '16884184', '16884185', '16884186', '16884187', '16884188', '16884189', '16884190', '16884191', '16884192', '16884193', '16884194', '16884195', '16884196', '16884197', '16884203', '16884204', '16884205', '16884206', '16884212', '16884213', '16884214', '16884215', '16884216', '16884217', '16884218', '16884219', '16884220', '16884221', '16884222', '16884223', '16884224', '16884225', '16884226', '16884227', '16884228', '16884229', '16884230', '16884231', '16884207', '16884208', '16884209', '16884210', '16884211', '766'] 2024-03-07 03:36:24 proc_lsst.multi:146 [INFO] found job 16884232 in provider checkpoint proc_lsst.multi INFO: found job 16884232 in provider checkpoint 2024-03-07 03:36:24 proc_lsst.multi:201 [INFO] cancelling 16884232 on provider checkpoint proc_lsst.multi INFO: cancelling 16884232 on provider checkpoint 2024-03-07 03:36:24 proc_lsst.multi:146 [INFO] found job 16884233 in provider checkpoint proc_lsst.multi INFO: found job 16884233 in provider checkpoint 2024-03-07 03:36:24 proc_lsst.multi:201 [INFO] cancelling 16884233 on provider checkpoint proc_lsst.multi INFO: cancelling 16884233 on provider checkpoint 2024-03-07 03:36:24 proc_lsst.multi:146 [INFO] found job 16884234 in provider checkpoint proc_lsst.multi INFO: found job 16884234 in provider checkpoint 2024-03-07 03:36:24 proc_lsst.multi:201 [INFO] cancelling 16884234 on provider checkpoint proc_lsst.multi INFO: cancelling 16884234 on provider checkpoint 2024-03-07 03:36:24 proc_lsst.multi:146 [INFO] found job 16884235 in provider checkpoint proc_lsst.multi INFO: found job 16884235 in provider checkpoint 2024-03-07 03:36:24 proc_lsst.multi:201 [INFO] cancelling 16884235 on provider checkpoint proc_lsst.multi INFO: cancelling 16884235 on provider checkpoint 2024-03-07 03:36:24 proc_lsst.multi:146 [INFO] found job 16884236 in provider checkpoint proc_lsst.multi INFO: found job 16884236 in provider checkpoint 2024-03-07 03:36:24 proc_lsst.multi:201 [INFO] cancelling 16884236 on provider checkpoint proc_lsst.multi INFO: cancelling 16884236 on provider checkpoint 2024-03-07 03:36:24 proc_lsst.multi:146 [INFO] found job 16884237 in provider checkpoint proc_lsst.multi INFO: found job 16884237 in provider checkpoint 2024-03-07 03:36:24 proc_lsst.multi:201 [INFO] cancelling 16884237 on provider checkpoint proc_lsst.multi INFO: cancelling 16884237 on provider checkpoint 2024-03-07 03:36:24 proc_lsst.multi:146 [INFO] found job 16884172 in provider astro proc_lsst.multi INFO: found job 16884172 in provider astro 2024-03-07 03:36:24 proc_lsst.multi:201 [INFO] cancelling 16884172 on provider astro proc_lsst.multi INFO: cancelling 16884172 on provider astro 2024-03-07 03:36:24 proc_lsst.multi:146 [INFO] found job 16884173 in provider astro proc_lsst.multi INFO: found job 16884173 in provider astro 2024-03-07 03:36:24 proc_lsst.multi:201 [INFO] cancelling 16884173 on provider astro proc_lsst.multi INFO: cancelling 16884173 on provider astro 2024-03-07 03:36:24 proc_lsst.multi:146 [INFO] found job 16884174 in provider astro proc_lsst.multi INFO: found job 16884174 in provider astro 2024-03-07 03:36:24 proc_lsst.multi:201 [INFO] cancelling 16884174 on provider astro proc_lsst.multi INFO: cancelling 16884174 on provider astro 2024-03-07 03:36:25 proc_lsst.multi:146 [INFO] found job 16884175 in provider astro proc_lsst.multi INFO: found job 16884175 in provider astro 2024-03-07 03:36:25 proc_lsst.multi:201 [INFO] cancelling 16884175 on provider astro proc_lsst.multi INFO: cancelling 16884175 on provider astro 2024-03-07 03:36:25 proc_lsst.multi:146 [INFO] found job 16884176 in provider astro proc_lsst.multi INFO: found job 16884176 in provider astro 2024-03-07 03:36:25 proc_lsst.multi:201 [INFO] cancelling 16884176 on provider astro proc_lsst.multi INFO: cancelling 16884176 on provider astro 2024-03-07 03:36:25 proc_lsst.multi:146 [INFO] found job 16884177 in provider astro proc_lsst.multi INFO: found job 16884177 in provider astro 2024-03-07 03:36:25 proc_lsst.multi:201 [INFO] cancelling 16884177 on provider astro proc_lsst.multi INFO: cancelling 16884177 on provider astro 2024-03-07 03:36:25 proc_lsst.multi:146 [INFO] found job 16884178 in provider astro proc_lsst.multi INFO: found job 16884178 in provider astro 2024-03-07 03:36:25 proc_lsst.multi:201 [INFO] cancelling 16884178 on provider astro proc_lsst.multi INFO: cancelling 16884178 on provider astro 2024-03-07 03:36:25 proc_lsst.multi:146 [INFO] found job 16884179 in provider astro proc_lsst.multi INFO: found job 16884179 in provider astro 2024-03-07 03:36:25 proc_lsst.multi:201 [INFO] cancelling 16884179 on provider astro proc_lsst.multi INFO: cancelling 16884179 on provider astro 2024-03-07 03:36:25 proc_lsst.multi:146 [INFO] found job 16884180 in provider astro proc_lsst.multi INFO: found job 16884180 in provider astro 2024-03-07 03:36:25 proc_lsst.multi:201 [INFO] cancelling 16884180 on provider astro proc_lsst.multi INFO: cancelling 16884180 on provider astro 2024-03-07 03:36:25 proc_lsst.multi:146 [INFO] found job 16884181 in provider astro proc_lsst.multi INFO: found job 16884181 in provider astro 2024-03-07 03:36:25 proc_lsst.multi:201 [INFO] cancelling 16884181 on provider astro proc_lsst.multi INFO: cancelling 16884181 on provider astro 2024-03-07 03:36:25 proc_lsst.multi:146 [INFO] found job 16884182 in provider astro proc_lsst.multi INFO: found job 16884182 in provider astro 2024-03-07 03:36:25 proc_lsst.multi:201 [INFO] cancelling 16884182 on provider astro proc_lsst.multi INFO: cancelling 16884182 on provider astro 2024-03-07 03:36:25 proc_lsst.multi:146 [INFO] found job 16884183 in provider astro proc_lsst.multi INFO: found job 16884183 in provider astro 2024-03-07 03:36:25 proc_lsst.multi:201 [INFO] cancelling 16884183 on provider astro proc_lsst.multi INFO: cancelling 16884183 on provider astro 2024-03-07 03:36:26 proc_lsst.multi:146 [INFO] found job 16884184 in provider astro proc_lsst.multi INFO: found job 16884184 in provider astro 2024-03-07 03:36:26 proc_lsst.multi:201 [INFO] cancelling 16884184 on provider astro proc_lsst.multi INFO: cancelling 16884184 on provider astro 2024-03-07 03:36:26 proc_lsst.multi:146 [INFO] found job 16884185 in provider astro proc_lsst.multi INFO: found job 16884185 in provider astro 2024-03-07 03:36:26 proc_lsst.multi:201 [INFO] cancelling 16884185 on provider astro proc_lsst.multi INFO: cancelling 16884185 on provider astro 2024-03-07 03:36:26 proc_lsst.multi:146 [INFO] found job 16884186 in provider astro proc_lsst.multi INFO: found job 16884186 in provider astro 2024-03-07 03:36:26 proc_lsst.multi:201 [INFO] cancelling 16884186 on provider astro proc_lsst.multi INFO: cancelling 16884186 on provider astro 2024-03-07 03:36:26 proc_lsst.multi:146 [INFO] found job 16884187 in provider astro proc_lsst.multi INFO: found job 16884187 in provider astro 2024-03-07 03:36:26 proc_lsst.multi:201 [INFO] cancelling 16884187 on provider astro proc_lsst.multi INFO: cancelling 16884187 on provider astro 2024-03-07 03:36:26 proc_lsst.multi:146 [INFO] found job 16884188 in provider astro proc_lsst.multi INFO: found job 16884188 in provider astro 2024-03-07 03:36:26 proc_lsst.multi:201 [INFO] cancelling 16884188 on provider astro proc_lsst.multi INFO: cancelling 16884188 on provider astro 2024-03-07 03:36:26 proc_lsst.multi:146 [INFO] found job 16884189 in provider astro proc_lsst.multi INFO: found job 16884189 in provider astro 2024-03-07 03:36:26 proc_lsst.multi:201 [INFO] cancelling 16884189 on provider astro proc_lsst.multi INFO: cancelling 16884189 on provider astro 2024-03-07 03:36:26 proc_lsst.multi:146 [INFO] found job 16884190 in provider astro proc_lsst.multi INFO: found job 16884190 in provider astro 2024-03-07 03:36:26 proc_lsst.multi:201 [INFO] cancelling 16884190 on provider astro proc_lsst.multi INFO: cancelling 16884190 on provider astro 2024-03-07 03:36:26 proc_lsst.multi:146 [INFO] found job 16884191 in provider astro proc_lsst.multi INFO: found job 16884191 in provider astro 2024-03-07 03:36:26 proc_lsst.multi:201 [INFO] cancelling 16884191 on provider astro proc_lsst.multi INFO: cancelling 16884191 on provider astro 2024-03-07 03:36:26 proc_lsst.multi:146 [INFO] found job 16884192 in provider astro proc_lsst.multi INFO: found job 16884192 in provider astro 2024-03-07 03:36:26 proc_lsst.multi:201 [INFO] cancelling 16884192 on provider astro proc_lsst.multi INFO: cancelling 16884192 on provider astro 2024-03-07 03:36:26 proc_lsst.multi:146 [INFO] found job 16884193 in provider astro proc_lsst.multi INFO: found job 16884193 in provider astro 2024-03-07 03:36:26 proc_lsst.multi:201 [INFO] cancelling 16884193 on provider astro proc_lsst.multi INFO: cancelling 16884193 on provider astro 2024-03-07 03:36:27 proc_lsst.multi:146 [INFO] found job 16884194 in provider astro proc_lsst.multi INFO: found job 16884194 in provider astro 2024-03-07 03:36:27 proc_lsst.multi:201 [INFO] cancelling 16884194 on provider astro proc_lsst.multi INFO: cancelling 16884194 on provider astro 2024-03-07 03:36:27 proc_lsst.multi:146 [INFO] found job 16884195 in provider astro proc_lsst.multi INFO: found job 16884195 in provider astro 2024-03-07 03:36:27 proc_lsst.multi:201 [INFO] cancelling 16884195 on provider astro proc_lsst.multi INFO: cancelling 16884195 on provider astro 2024-03-07 03:36:27 proc_lsst.multi:146 [INFO] found job 16884196 in provider astro proc_lsst.multi INFO: found job 16884196 in provider astro 2024-03-07 03:36:27 proc_lsst.multi:201 [INFO] cancelling 16884196 on provider astro proc_lsst.multi INFO: cancelling 16884196 on provider astro 2024-03-07 03:36:27 proc_lsst.multi:146 [INFO] found job 16884197 in provider astro proc_lsst.multi INFO: found job 16884197 in provider astro 2024-03-07 03:36:27 proc_lsst.multi:201 [INFO] cancelling 16884197 on provider astro proc_lsst.multi INFO: cancelling 16884197 on provider astro 2024-03-07 03:36:27 proc_lsst.multi:146 [INFO] found job 16884203 in provider astro proc_lsst.multi INFO: found job 16884203 in provider astro 2024-03-07 03:36:27 proc_lsst.multi:201 [INFO] cancelling 16884203 on provider astro proc_lsst.multi INFO: cancelling 16884203 on provider astro 2024-03-07 03:36:27 proc_lsst.multi:146 [INFO] found job 16884204 in provider astro proc_lsst.multi INFO: found job 16884204 in provider astro 2024-03-07 03:36:27 proc_lsst.multi:201 [INFO] cancelling 16884204 on provider astro proc_lsst.multi INFO: cancelling 16884204 on provider astro 2024-03-07 03:36:27 proc_lsst.multi:146 [INFO] found job 16884205 in provider astro proc_lsst.multi INFO: found job 16884205 in provider astro 2024-03-07 03:36:27 proc_lsst.multi:201 [INFO] cancelling 16884205 on provider astro proc_lsst.multi INFO: cancelling 16884205 on provider astro 2024-03-07 03:36:27 proc_lsst.multi:146 [INFO] found job 16884206 in provider astro proc_lsst.multi INFO: found job 16884206 in provider astro 2024-03-07 03:36:27 proc_lsst.multi:201 [INFO] cancelling 16884206 on provider astro proc_lsst.multi INFO: cancelling 16884206 on provider astro 2024-03-07 03:36:27 proc_lsst.multi:146 [INFO] found job 16884212 in provider escience_a40 proc_lsst.multi INFO: found job 16884212 in provider escience_a40 2024-03-07 03:36:27 proc_lsst.multi:201 [INFO] cancelling 16884212 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884212 on provider escience_a40 2024-03-07 03:36:28 proc_lsst.multi:146 [INFO] found job 16884213 in provider escience_a40 proc_lsst.multi INFO: found job 16884213 in provider escience_a40 2024-03-07 03:36:28 proc_lsst.multi:201 [INFO] cancelling 16884213 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884213 on provider escience_a40 2024-03-07 03:36:28 proc_lsst.multi:146 [INFO] found job 16884214 in provider escience_a40 proc_lsst.multi INFO: found job 16884214 in provider escience_a40 2024-03-07 03:36:28 proc_lsst.multi:201 [INFO] cancelling 16884214 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884214 on provider escience_a40 2024-03-07 03:36:28 proc_lsst.multi:146 [INFO] found job 16884215 in provider escience_a40 proc_lsst.multi INFO: found job 16884215 in provider escience_a40 2024-03-07 03:36:28 proc_lsst.multi:201 [INFO] cancelling 16884215 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884215 on provider escience_a40 2024-03-07 03:36:28 proc_lsst.multi:146 [INFO] found job 16884216 in provider escience_a40 proc_lsst.multi INFO: found job 16884216 in provider escience_a40 2024-03-07 03:36:28 proc_lsst.multi:201 [INFO] cancelling 16884216 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884216 on provider escience_a40 2024-03-07 03:36:28 proc_lsst.multi:146 [INFO] found job 16884217 in provider escience_a40 proc_lsst.multi INFO: found job 16884217 in provider escience_a40 2024-03-07 03:36:28 proc_lsst.multi:201 [INFO] cancelling 16884217 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884217 on provider escience_a40 2024-03-07 03:36:28 proc_lsst.multi:146 [INFO] found job 16884218 in provider escience_a40 proc_lsst.multi INFO: found job 16884218 in provider escience_a40 2024-03-07 03:36:28 proc_lsst.multi:201 [INFO] cancelling 16884218 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884218 on provider escience_a40 2024-03-07 03:36:28 proc_lsst.multi:146 [INFO] found job 16884219 in provider escience_a40 proc_lsst.multi INFO: found job 16884219 in provider escience_a40 2024-03-07 03:36:28 proc_lsst.multi:201 [INFO] cancelling 16884219 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884219 on provider escience_a40 2024-03-07 03:36:28 proc_lsst.multi:146 [INFO] found job 16884220 in provider escience_a40 proc_lsst.multi INFO: found job 16884220 in provider escience_a40 2024-03-07 03:36:28 proc_lsst.multi:201 [INFO] cancelling 16884220 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884220 on provider escience_a40 2024-03-07 03:36:28 proc_lsst.multi:146 [INFO] found job 16884221 in provider escience_a40 proc_lsst.multi INFO: found job 16884221 in provider escience_a40 2024-03-07 03:36:28 proc_lsst.multi:201 [INFO] cancelling 16884221 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884221 on provider escience_a40 2024-03-07 03:36:28 proc_lsst.multi:146 [INFO] found job 16884222 in provider escience_a40 proc_lsst.multi INFO: found job 16884222 in provider escience_a40 2024-03-07 03:36:28 proc_lsst.multi:201 [INFO] cancelling 16884222 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884222 on provider escience_a40 2024-03-07 03:36:28 proc_lsst.multi:146 [INFO] found job 16884223 in provider escience_a40 proc_lsst.multi INFO: found job 16884223 in provider escience_a40 2024-03-07 03:36:28 proc_lsst.multi:201 [INFO] cancelling 16884223 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884223 on provider escience_a40 2024-03-07 03:36:28 proc_lsst.multi:146 [INFO] found job 16884224 in provider escience_a40 proc_lsst.multi INFO: found job 16884224 in provider escience_a40 2024-03-07 03:36:28 proc_lsst.multi:201 [INFO] cancelling 16884224 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884224 on provider escience_a40 2024-03-07 03:36:29 proc_lsst.multi:146 [INFO] found job 16884225 in provider escience_a40 proc_lsst.multi INFO: found job 16884225 in provider escience_a40 2024-03-07 03:36:29 proc_lsst.multi:201 [INFO] cancelling 16884225 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884225 on provider escience_a40 2024-03-07 03:36:29 proc_lsst.multi:146 [INFO] found job 16884226 in provider escience_a40 proc_lsst.multi INFO: found job 16884226 in provider escience_a40 2024-03-07 03:36:29 proc_lsst.multi:201 [INFO] cancelling 16884226 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884226 on provider escience_a40 2024-03-07 03:36:29 proc_lsst.multi:146 [INFO] found job 16884227 in provider escience_a40 proc_lsst.multi INFO: found job 16884227 in provider escience_a40 2024-03-07 03:36:29 proc_lsst.multi:201 [INFO] cancelling 16884227 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884227 on provider escience_a40 2024-03-07 03:36:29 proc_lsst.multi:146 [INFO] found job 16884228 in provider escience_a40 proc_lsst.multi INFO: found job 16884228 in provider escience_a40 2024-03-07 03:36:29 proc_lsst.multi:201 [INFO] cancelling 16884228 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884228 on provider escience_a40 2024-03-07 03:36:29 proc_lsst.multi:146 [INFO] found job 16884229 in provider escience_a40 proc_lsst.multi INFO: found job 16884229 in provider escience_a40 2024-03-07 03:36:29 proc_lsst.multi:201 [INFO] cancelling 16884229 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884229 on provider escience_a40 2024-03-07 03:36:29 proc_lsst.multi:146 [INFO] found job 16884230 in provider escience_a40 proc_lsst.multi INFO: found job 16884230 in provider escience_a40 2024-03-07 03:36:29 proc_lsst.multi:201 [INFO] cancelling 16884230 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884230 on provider escience_a40 2024-03-07 03:36:29 proc_lsst.multi:146 [INFO] found job 16884231 in provider escience_a40 proc_lsst.multi INFO: found job 16884231 in provider escience_a40 2024-03-07 03:36:29 proc_lsst.multi:201 [INFO] cancelling 16884231 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884231 on provider escience_a40 2024-03-07 03:36:29 proc_lsst.multi:146 [INFO] found job 16884207 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884207 in provider escience_rtx6k 2024-03-07 03:36:29 proc_lsst.multi:201 [INFO] cancelling 16884207 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16884207 on provider escience_rtx6k 2024-03-07 03:36:29 proc_lsst.multi:146 [INFO] found job 16884208 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884208 in provider escience_rtx6k 2024-03-07 03:36:29 proc_lsst.multi:201 [INFO] cancelling 16884208 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16884208 on provider escience_rtx6k 2024-03-07 03:36:29 proc_lsst.multi:146 [INFO] found job 16884209 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884209 in provider escience_rtx6k 2024-03-07 03:36:29 proc_lsst.multi:201 [INFO] cancelling 16884209 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16884209 on provider escience_rtx6k 2024-03-07 03:36:29 proc_lsst.multi:146 [INFO] found job 16884210 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884210 in provider escience_rtx6k 2024-03-07 03:36:29 proc_lsst.multi:201 [INFO] cancelling 16884210 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16884210 on provider escience_rtx6k 2024-03-07 03:36:30 proc_lsst.multi:146 [INFO] found job 16884211 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884211 in provider escience_rtx6k 2024-03-07 03:36:30 proc_lsst.multi:201 [INFO] cancelling 16884211 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16884211 on provider escience_rtx6k 2024-03-07 03:36:30 proc_lsst.multi:146 [INFO] found job 766 in provider local proc_lsst.multi INFO: found job 766 in provider local 2024-03-07 03:36:30 proc_lsst.multi:201 [INFO] cancelling 766 on provider local proc_lsst.multi INFO: cancelling 766 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 766 parsl.providers.local.local WARNING: Failed to kill PID: 766 and child processes on local 2024-03-07 03:36:31 proc_lsst.multi:50 [INFO] no new jobs since last cancel, resuming executor shutdown proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt parsl.dataflow.dflow INFO: Shut down executor multi parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete parsl.executors.threads DEBUG: Done with executor shutdown parsl.dataflow.dflow INFO: Shut down executor _parsl_internal parsl.dataflow.dflow INFO: Terminated executors parsl.dataflow.dflow INFO: DFK cleanup complete parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 165.1345 seconds lsst.ctrl.bps.drivers INFO: Run 'DEEP_20220821_B1m_science#step0_20240307T033310Z' submitted for execution with id 'None' lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 165.1407 seconds; current memory usage: 0.273 Gibyte, delta: 0.001 Gibyte, peak delta: 0.003 Gibyte lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 195.5236 seconds; current memory usage: 0.273 Gibyte, delta: 0.136 Gibyte, peak delta: 0.138 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.275 Gibyte (main), 0.274 Gibyte (largest child process) Run Id: None Run Name: DEEP_20220821_B1m_science#step0_20240307T033310Z parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up