+ PROC_LSST_SUBMIT_PATH=/mmfs1/home/stevengs/dirac/DEEP/submit + bps submit /gscratch/dirac/shared/opt/proc_lsst/pipelines/submit.yaml -b /mmfs1/home/stevengs/dirac/DEEP/repo -i DEEP/20190601/A1c --output-run DEEP/20190601/A1c/science#step0/20240310T013701Z --qgraph pipeline.qgraph lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process. lsst.ctrl.bps.drivers INFO: Starting submission process lsst.ctrl.bps.drivers INFO: Initializing execution environment lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 8.7521 seconds; current memory usage: 0.177 Gibyte, delta: 0.041 Gibyte, peak delta: 0.054 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.190 Gibyte (main), 0.000 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph) lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.5165 seconds lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.5047 seconds lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/pipeline_orig.qgraph DEEP/20190601/A1c/science#step0/20240310T013701Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/pipeline.qgraph lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 3.7451 seconds lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 11.2107 seconds; current memory usage: 0.270 Gibyte, delta: 0.092 Gibyte, peak delta: 0.080 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.270 Gibyte (main), 0.260 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs) lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0170 seconds; current memory usage: 0.270 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.270 Gibyte (main), 0.260 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 186 cluster(s) lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow) lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20190601_A1c_science#step0_20240310T013701Z' lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 0.6013 seconds; current memory usage: 0.270 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.270 Gibyte (main), 0.260 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 188 job(s) (including final) lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow) parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2 Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses s_addresses.add(address_by_interface(interface)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper retval = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface return socket.inet_ntoa(fcntl.ioctl( ^^^^^^^^^^^^ OSError: [Errno 99] Cannot assign requested address lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 1.6949 seconds; current memory usage: 0.271 Gibyte, delta: 0.002 Gibyte, peak delta: 0.002 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.271 Gibyte (main), 0.260 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting submit stage lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000 parsl.dataflow.dflow INFO: Starting DataFlowKernel with config Config( app_cache=True, checkpoint_files=None, checkpoint_mode='task_exit', checkpoint_period=None, executors=[MultiHighThroughputExecutor()], garbage_collect=True, initialize_logging=True, internal_tasks_max_threads=10, max_idletime=120.0, monitoring=None, retries=1, retry_handler=None, run_dir='runinfo', strategy='simple', usage_tracking=False ) parsl.dataflow.dflow INFO: Parsl version: 2023.06.12 parsl.usage_tracking.usage DEBUG: Tracking status: False parsl.dataflow.dflow INFO: Run id is: 8d086dea-2c6b-44e3-b5ab-ef11f9d2184d parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name parsl.dataflow.memoization INFO: App caching initialized parsl.dataflow.strategy DEBUG: Scaling strategy: simple parsl.executors.high_throughput.executor DEBUG: Starting queue management thread parsl.executors.high_throughput.executor DEBUG: queue management worker starting parsl.executors.high_throughput.executor DEBUG: Started queue management thread Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z parsl.executors.high_throughput.executor DEBUG: Created management thread: parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id={block_id} --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider: parsl.executors.status_handling INFO: Scaling out by 1 blocks parsl.executors.status_handling INFO: Allocated block ID 0 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0 2024-03-10 01:37:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 2024-03-10 01:37:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:37:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-10 01:37:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:37:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local parsl.providers.local.local DEBUG: Launching in remote mode 2024-03-10 01:37:44 proc_lsst.multi:170 [INFO] [multi] job_id 44401 proc_lsst.multi INFO: [multi] job_id 44401 2024-03-10 01:37:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:37:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:37:44 proc_lsst.multi:178 [INFO] [multi] provider local accepted submit and returned 44401 proc_lsst.multi INFO: [multi] provider local accepted submit and returned 44401 parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 44401 parsl.dataflow.job_status_poller DEBUG: Adding executor multi parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 0 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 0 has memoization hash 9a5c6f0b1599d331ce367d16734b9b9a parsl.dataflow.memoization INFO: Task 0 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105f5f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1 parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/84c2d1c4-9e10-43af-a259-94ad110d4c65_isrForCrosstalkSources_860803_44.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/84c2d1c4-9e10-43af-a259-94ad110d4c65_isrForCrosstalkSources_860803_44.stderr parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1 has memoization hash c5ba040057a58696c4253efd35e8797d parsl.dataflow.memoization INFO: Task 1 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105f5f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 2 parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/92976e06-ac94-40e7-8e53-5e34106bd6a3_isrForCrosstalkSources_860801_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/92976e06-ac94-40e7-8e53-5e34106bd6a3_isrForCrosstalkSources_860801_49.stderr parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2 has memoization hash 30188158eb8e6e6e6980e6194255a1e1 parsl.dataflow.memoization INFO: Task 2 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105f5fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 3 parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/11b36829-3f8c-4888-bfbe-254c3b6421bb_isrForCrosstalkSources_860803_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/11b36829-3f8c-4888-bfbe-254c3b6421bb_isrForCrosstalkSources_860803_8.stderr parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3 has memoization hash f3666efbbf11bbba35565bd0f2507f54 parsl.dataflow.memoization INFO: Task 3 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105f5f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 4 parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/8d557ce3-135a-4574-ae31-b3b20698843f_isrForCrosstalkSources_860803_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/8d557ce3-135a-4574-ae31-b3b20698843f_isrForCrosstalkSources_860803_42.stderr parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4 has memoization hash 5718219c26222edd42d4d7b93a79d6fa parsl.dataflow.memoization INFO: Task 4 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105f5fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 5 parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/5b12376e-5c99-459c-9034-e707294ef90d_isrForCrosstalkSources_860802_28.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/5b12376e-5c99-459c-9034-e707294ef90d_isrForCrosstalkSources_860802_28.stderr parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5 has memoization hash f478d9dce681723d2e4fe44147c1c1d0 parsl.dataflow.memoization INFO: Task 5 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105f5fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 6 parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/1cabc528-c495-4004-ae61-f26f33ccd1dd_isrForCrosstalkSources_860803_47.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/1cabc528-c495-4004-ae61-f26f33ccd1dd_isrForCrosstalkSources_860803_47.stderr parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 6 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 6 has memoization hash e40d4dceccdbaf27a477b17de7548969 parsl.dataflow.memoization INFO: Task 6 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105f5f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 7 parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/69ab5d8b-3c5d-4296-a84f-d9178bd6c13c_isrForCrosstalkSources_860802_42.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/69ab5d8b-3c5d-4296-a84f-d9178bd6c13c_isrForCrosstalkSources_860802_42.stderr parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 7 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 7 has memoization hash dba9d4c981b8db89270f7dd1fd8449e0 parsl.dataflow.memoization INFO: Task 7 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105f5fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 8 parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/8017b810-b22f-462c-82e1-52addb06ba8c_isrForCrosstalkSources_860803_5.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/8017b810-b22f-462c-82e1-52addb06ba8c_isrForCrosstalkSources_860803_5.stderr parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 8 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 8 has memoization hash 95627e28428e55d36da7d66f22d9aeb5 parsl.dataflow.memoization INFO: Task 8 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105f5ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 9 parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/2e20cfae-845e-4f5e-ae0b-8364d1ab7914_isrForCrosstalkSources_860802_35.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/2e20cfae-845e-4f5e-ae0b-8364d1ab7914_isrForCrosstalkSources_860802_35.stderr parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 9 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 9 has memoization hash 79bdd93a1ccb72697a1e02d348048f1c parsl.dataflow.memoization INFO: Task 9 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105f5f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 10 parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/0ef0e551-bc62-47fa-a6f4-207e31e25040_isrForCrosstalkSources_860802_55.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/0ef0e551-bc62-47fa-a6f4-207e31e25040_isrForCrosstalkSources_860802_55.stderr parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 10 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 10 has memoization hash f01a52fb556210f11540184fb14af3be parsl.dataflow.memoization INFO: Task 10 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105f5fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 11 parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/27f97124-4191-4a9e-9332-a8b519369327_isrForCrosstalkSources_860802_60.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/27f97124-4191-4a9e-9332-a8b519369327_isrForCrosstalkSources_860802_60.stderr parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 11 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 11 has memoization hash 2a5af4add3cd845dedb14f69d1c42f08 parsl.dataflow.memoization INFO: Task 11 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105f5fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 12 parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/fe4e406a-f54a-466e-b94c-cc3babe3ea73_isrForCrosstalkSources_860803_22.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/fe4e406a-f54a-466e-b94c-cc3babe3ea73_isrForCrosstalkSources_860803_22.stderr parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 12 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 12 has memoization hash 5b5d4022490d69ffa149d5aa1a09cfba parsl.dataflow.memoization INFO: Task 12 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa00e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 13 parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/05bb4e71-5687-466f-8361-2caefd851eab_isrForCrosstalkSources_860803_34.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/05bb4e71-5687-466f-8361-2caefd851eab_isrForCrosstalkSources_860803_34.stderr parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 13 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 13 has memoization hash 2b320d1b8465d42fe1b35e6e0550fee8 parsl.dataflow.memoization INFO: Task 13 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa0400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 14 parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/7855529c-dc41-4801-ac53-c497ecc159ff_isrForCrosstalkSources_860803_9.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/7855529c-dc41-4801-ac53-c497ecc159ff_isrForCrosstalkSources_860803_9.stderr parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 14 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 14 has memoization hash 210752207651ea9904ad9c734e8d9952 parsl.dataflow.memoization INFO: Task 14 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa05e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 15 parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/f61c3c66-643a-4e18-a16c-1cf82e6afd29_isrForCrosstalkSources_860801_41.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/f61c3c66-643a-4e18-a16c-1cf82e6afd29_isrForCrosstalkSources_860801_41.stderr parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 15 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 15 has memoization hash 427be0ce20ee88a02a56f281c51c669b parsl.dataflow.memoization INFO: Task 15 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa02c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 16 parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/f0c55dbf-7437-49b6-bdeb-f5ddcbf19439_isrForCrosstalkSources_860803_4.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/f0c55dbf-7437-49b6-bdeb-f5ddcbf19439_isrForCrosstalkSources_860803_4.stderr parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 16 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 16 has memoization hash 749f285c787b279f62d515d4b0c1139d parsl.dataflow.memoization INFO: Task 16 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 17 parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/9191625d-335c-4e77-b09d-5e0f115aca4f_isrForCrosstalkSources_860801_13.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/9191625d-335c-4e77-b09d-5e0f115aca4f_isrForCrosstalkSources_860801_13.stderr parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 17 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 17 has memoization hash 95143b39cccd2b5dfa77332aec33dfd9 parsl.dataflow.memoization INFO: Task 17 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa04a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 18 parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/69388045-56f7-4cb7-8ecf-0f695e5b805d_isrForCrosstalkSources_860802_11.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/69388045-56f7-4cb7-8ecf-0f695e5b805d_isrForCrosstalkSources_860802_11.stderr parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 18 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 18 has memoization hash 326d7340e218c8770a5956c77615134b parsl.dataflow.memoization INFO: Task 18 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa0720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 19 parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/d2338ba0-7760-402f-8b22-d31cbbbc9285_isrForCrosstalkSources_860801_5.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/d2338ba0-7760-402f-8b22-d31cbbbc9285_isrForCrosstalkSources_860801_5.stderr parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 19 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 19 has memoization hash d396bfe43cfbcef2dc70a48f472e7766 parsl.dataflow.memoization INFO: Task 19 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa07c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 20 parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/7c26a2fa-f35f-4a3f-9286-dbdf28f33e1b_isrForCrosstalkSources_860801_23.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/7c26a2fa-f35f-4a3f-9286-dbdf28f33e1b_isrForCrosstalkSources_860801_23.stderr parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 20 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 20 has memoization hash bf630881a8235ba197368a3386fb2202 parsl.dataflow.memoization INFO: Task 20 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa0860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 21 parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/18b9e2b8-cf10-4ef6-858e-74317578427c_isrForCrosstalkSources_860801_7.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/18b9e2b8-cf10-4ef6-858e-74317578427c_isrForCrosstalkSources_860801_7.stderr parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 21 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 21 has memoization hash ca7b7bf53a084d55391e9fb834098610 parsl.dataflow.memoization INFO: Task 21 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa09a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 22 parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/5a0d6b15-e893-4565-af8d-8f9fab96aaf4_isrForCrosstalkSources_860802_62.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/5a0d6b15-e893-4565-af8d-8f9fab96aaf4_isrForCrosstalkSources_860802_62.stderr parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 22 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash d1e2fa1b5e318d7a746996f48bf48f08 parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa0900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 23 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/5c609f61-cae9-4b03-8b32-2b76452a6404_isrForCrosstalkSources_860801_62.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/5c609f61-cae9-4b03-8b32-2b76452a6404_isrForCrosstalkSources_860801_62.stderr parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 23 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 23 has memoization hash a0cf734a078bf12c8b36c84c15ae9290 parsl.dataflow.memoization INFO: Task 23 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa0a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 23 try 0 launched on executor multi with executor id 24 parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/5b1052a3-2b2b-48c9-a923-003bd40699d7_isrForCrosstalkSources_860802_1.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/5b1052a3-2b2b-48c9-a923-003bd40699d7_isrForCrosstalkSources_860802_1.stderr parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 24 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 24 has memoization hash 0fad56391e23cfa49d656b9c145b17b9 parsl.dataflow.memoization INFO: Task 24 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa0ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 25 parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/ced5c32d-8c38-4cb9-8640-85765b3fd23f_isrForCrosstalkSources_860803_12.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/ced5c32d-8c38-4cb9-8640-85765b3fd23f_isrForCrosstalkSources_860803_12.stderr parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 25 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 25 has memoization hash b85694b0fcf7b5f58a8f995475e5b812 parsl.dataflow.memoization INFO: Task 25 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 26 parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/72b58ae1-bc9b-4472-97e3-15487ab9428f_isrForCrosstalkSources_860802_47.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/72b58ae1-bc9b-4472-97e3-15487ab9428f_isrForCrosstalkSources_860802_47.stderr parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 26 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 26 has memoization hash 432d9f0abd9a51a3fbe30d1c692f18d3 parsl.dataflow.memoization INFO: Task 26 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa0cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 27 parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/c41fa473-f795-4cab-bf40-8ba0a6618ab1_isrForCrosstalkSources_860803_14.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/c41fa473-f795-4cab-bf40-8ba0a6618ab1_isrForCrosstalkSources_860803_14.stderr parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 27 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 27 has memoization hash d45d5ad6882089137554410f081dab2a parsl.dataflow.memoization INFO: Task 27 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa0c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 28 parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/a7334b40-6503-485d-9143-123435c02d2d_isrForCrosstalkSources_860801_43.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/a7334b40-6503-485d-9143-123435c02d2d_isrForCrosstalkSources_860801_43.stderr parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 28 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash a4a2325c17666388e2d2e639ca2b70ca parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa0d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 29 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/7e989704-dc7f-4d0f-9432-ea1c30d727af_isrForCrosstalkSources_860801_11.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/7e989704-dc7f-4d0f-9432-ea1c30d727af_isrForCrosstalkSources_860801_11.stderr parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 29 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 29 has memoization hash 94cc5b62a9f97de53824d89752c57344 parsl.dataflow.memoization INFO: Task 29 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa0e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 29 try 0 launched on executor multi with executor id 30 parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/6396ea2b-ea74-45d5-bc1c-df8445a13d86_isrForCrosstalkSources_860803_54.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/6396ea2b-ea74-45d5-bc1c-df8445a13d86_isrForCrosstalkSources_860803_54.stderr parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 30 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 30 has memoization hash 575a9f20e622e82ba21fc988298a77c3 parsl.dataflow.memoization INFO: Task 30 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 31 parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/d2716afc-c266-4939-aa6c-b7d1ddd06c53_isrForCrosstalkSources_860803_15.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/d2716afc-c266-4939-aa6c-b7d1ddd06c53_isrForCrosstalkSources_860803_15.stderr parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 31 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 31 has memoization hash 93bfe4cf6db9e8e8b3a6c2f9a55a714d parsl.dataflow.memoization INFO: Task 31 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa0fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 32 parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/9eddd923-8e43-4f35-8ddc-32b8fe00e085_isrForCrosstalkSources_860801_45.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/9eddd923-8e43-4f35-8ddc-32b8fe00e085_isrForCrosstalkSources_860801_45.stderr parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 32 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 32 has memoization hash df01b2d10b1a1404b86678b006c53d67 parsl.dataflow.memoization INFO: Task 32 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa0f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 33 parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/652b21af-a4cd-427d-b30e-85e5156a7a54_isrForCrosstalkSources_860801_27.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/652b21af-a4cd-427d-b30e-85e5156a7a54_isrForCrosstalkSources_860801_27.stderr parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 33 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 33 has memoization hash 02b7dc4f0d39d1b20c1f40a5778865b9 parsl.dataflow.memoization INFO: Task 33 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 34 parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/27e9085d-e0b4-4154-9262-b353b57807ac_isrForCrosstalkSources_860801_1.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/27e9085d-e0b4-4154-9262-b353b57807ac_isrForCrosstalkSources_860801_1.stderr parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 34 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash 1caf9f5268c8dd823c1cb45ea803beed parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa1120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 35 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/37f2f695-b247-48d1-8b33-76469229e1c2_isrForCrosstalkSources_860801_25.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/37f2f695-b247-48d1-8b33-76469229e1c2_isrForCrosstalkSources_860801_25.stderr parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 35 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 35 has memoization hash c489046b58a961f7b6b7822e453ad50a parsl.dataflow.memoization INFO: Task 35 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 35 try 0 launched on executor multi with executor id 36 parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/c9ec1f95-cd5d-46b7-a5f2-36d80dd66872_isrForCrosstalkSources_860801_26.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/c9ec1f95-cd5d-46b7-a5f2-36d80dd66872_isrForCrosstalkSources_860801_26.stderr parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 36 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 36 has memoization hash df8614665a08c9f9091ffb6f514feceb parsl.dataflow.memoization INFO: Task 36 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa1300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 37 parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/e1760c71-dcc5-469c-bded-8deb87623143_isrForCrosstalkSources_860801_44.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/e1760c71-dcc5-469c-bded-8deb87623143_isrForCrosstalkSources_860801_44.stderr parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 37 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 37 has memoization hash 1e887eb97406001014571832505b472d parsl.dataflow.memoization INFO: Task 37 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa1260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 38 parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/08668b3d-0289-4913-b965-6860481105c8_isrForCrosstalkSources_860803_45.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/08668b3d-0289-4913-b965-6860481105c8_isrForCrosstalkSources_860803_45.stderr parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 38 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 38 has memoization hash 9d06e3c8c0e6cf045308223b7969d4fa parsl.dataflow.memoization INFO: Task 38 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa13a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 39 parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/015b04f1-e758-4003-8841-ebfb60cbe44c_isrForCrosstalkSources_860803_56.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/015b04f1-e758-4003-8841-ebfb60cbe44c_isrForCrosstalkSources_860803_56.stderr parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 39 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 39 has memoization hash 2ff538ec2f78275562891cbdb40f323f parsl.dataflow.memoization INFO: Task 39 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa1440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 40 parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/2abf3afa-98f6-4ccc-80b2-a3e0d3d81bfb_isrForCrosstalkSources_860802_18.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/2abf3afa-98f6-4ccc-80b2-a3e0d3d81bfb_isrForCrosstalkSources_860802_18.stderr parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 40 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 40 has memoization hash 426dd8f023795c0d0df38ae46ff94918 parsl.dataflow.memoization INFO: Task 40 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa14e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 41 parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/51eb24ba-d807-42fb-a1ed-c2b3e0e9ac7b_isrForCrosstalkSources_860802_61.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/51eb24ba-d807-42fb-a1ed-c2b3e0e9ac7b_isrForCrosstalkSources_860802_61.stderr parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 41 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 41 has memoization hash ab415f3cf90801d0673e527736bfe5c3 parsl.dataflow.memoization INFO: Task 41 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa1620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 42 parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/b494bdd2-0a27-4e7e-aecc-90c23b64be55_isrForCrosstalkSources_860802_21.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/b494bdd2-0a27-4e7e-aecc-90c23b64be55_isrForCrosstalkSources_860802_21.stderr parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 42 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 42 has memoization hash f1991707abedc75f0f63389726438b7a parsl.dataflow.memoization INFO: Task 42 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa1580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 43 parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/44bc8a8e-b038-435d-a2e0-df5ed2749f7b_isrForCrosstalkSources_860801_31.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/44bc8a8e-b038-435d-a2e0-df5ed2749f7b_isrForCrosstalkSources_860801_31.stderr parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 43 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 43 has memoization hash b39de7e8e7970e4a644c11343c310996 parsl.dataflow.memoization INFO: Task 43 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 44 parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/10d0da8c-fb6c-418e-b160-446775441c01_isrForCrosstalkSources_860801_19.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/10d0da8c-fb6c-418e-b160-446775441c01_isrForCrosstalkSources_860801_19.stderr parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 44 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 44 has memoization hash ea78c28da6ad031223df889bc9cff126 parsl.dataflow.memoization INFO: Task 44 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa1760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 45 parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/c1368ff9-cb09-46c9-87a9-db1cb28d6801_isrForCrosstalkSources_860802_37.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/c1368ff9-cb09-46c9-87a9-db1cb28d6801_isrForCrosstalkSources_860802_37.stderr parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 45 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 45 has memoization hash 9d66429f7cafc7a81463373518cfb299 parsl.dataflow.memoization INFO: Task 45 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 46 parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/c6e088e1-25db-49e8-8cc3-4861acc12d0b_isrForCrosstalkSources_860801_42.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/c6e088e1-25db-49e8-8cc3-4861acc12d0b_isrForCrosstalkSources_860801_42.stderr parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 46 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 46 has memoization hash 0f3c8aecc01fc0e27234ec682e8d1a07 parsl.dataflow.memoization INFO: Task 46 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa1940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 47 parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/90d7d673-3f92-41c4-95a4-25d586f0483b_isrForCrosstalkSources_860802_12.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/90d7d673-3f92-41c4-95a4-25d586f0483b_isrForCrosstalkSources_860802_12.stderr parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 47 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 47 has memoization hash f2932e634f975f0fc873d5f724cf35b9 parsl.dataflow.memoization INFO: Task 47 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa18a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 48 parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/a1ccf741-a2ba-4004-af1f-a16c669b6a37_isrForCrosstalkSources_860801_17.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/a1ccf741-a2ba-4004-af1f-a16c669b6a37_isrForCrosstalkSources_860801_17.stderr parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 48 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 48 has memoization hash 98828376025df308d0c5556eecc3099f parsl.dataflow.memoization INFO: Task 48 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 49 parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/afb1d44f-f0f6-46df-9e5c-5bb2094e095d_isrForCrosstalkSources_860802_33.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/afb1d44f-f0f6-46df-9e5c-5bb2094e095d_isrForCrosstalkSources_860802_33.stderr parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 49 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 49 has memoization hash 8c8d10dcc258a5a94ab1b541bdfa4cef parsl.dataflow.memoization INFO: Task 49 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa1a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 50 parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/ba95771b-5284-4cf5-b7b8-0c0108fd4ea8_isrForCrosstalkSources_860802_26.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/ba95771b-5284-4cf5-b7b8-0c0108fd4ea8_isrForCrosstalkSources_860802_26.stderr parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 50 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 50 has memoization hash 636174db4a4a18da96147bcf1460fab7 parsl.dataflow.memoization INFO: Task 50 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 51 parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/d3e647e7-bd9e-48ee-aa6a-9f1af98422d9_isrForCrosstalkSources_860803_27.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/d3e647e7-bd9e-48ee-aa6a-9f1af98422d9_isrForCrosstalkSources_860803_27.stderr parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 51 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 51 has memoization hash b5eea09734fc5b621ca89fc31fc2dab4 parsl.dataflow.memoization INFO: Task 51 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa1c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 52 parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/e1cb704e-52b5-44f7-acf1-405ac03c4a93_isrForCrosstalkSources_860803_52.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/e1cb704e-52b5-44f7-acf1-405ac03c4a93_isrForCrosstalkSources_860803_52.stderr parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 52 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash 8b91f4adfb471798ec11824cb0934ac3 parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa1bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 53 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/0f68b69c-b2ad-4779-b81e-1b3182452cb8_isrForCrosstalkSources_860802_58.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/0f68b69c-b2ad-4779-b81e-1b3182452cb8_isrForCrosstalkSources_860802_58.stderr parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 53 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 53 has memoization hash 4d059948c2ee3d619d006fa05ad46e08 parsl.dataflow.memoization INFO: Task 53 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 53 try 0 launched on executor multi with executor id 54 parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/3a4e24f1-abec-400b-92c4-336b26aec61f_isrForCrosstalkSources_860801_3.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/3a4e24f1-abec-400b-92c4-336b26aec61f_isrForCrosstalkSources_860801_3.stderr parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 54 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 54 has memoization hash 5f1ef8f7b2f60889a47d2002d9e4e3fb parsl.dataflow.memoization INFO: Task 54 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa1da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 55 parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/c09087a2-69ec-419a-8a4e-006e8e242185_isrForCrosstalkSources_860801_56.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/c09087a2-69ec-419a-8a4e-006e8e242185_isrForCrosstalkSources_860801_56.stderr parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 55 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 55 has memoization hash 9a992622ffcef472317f7479af199a23 parsl.dataflow.memoization INFO: Task 55 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa1e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 56 parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/d62e3bea-8f52-4554-8a28-9b7e6e5659b6_isrForCrosstalkSources_860803_48.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/d62e3bea-8f52-4554-8a28-9b7e6e5659b6_isrForCrosstalkSources_860803_48.stderr parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 56 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 56 has memoization hash 90a38bdd8ea01ba81d3b58c645e85c46 parsl.dataflow.memoization INFO: Task 56 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa1f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 57 parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/d0f3281b-b9cb-46bc-adff-72180a402c77_isrForCrosstalkSources_860801_54.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/d0f3281b-b9cb-46bc-adff-72180a402c77_isrForCrosstalkSources_860801_54.stderr parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 57 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 57 has memoization hash 21f5b0a13f5a48207243f69a65a8ecb7 parsl.dataflow.memoization INFO: Task 57 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa1ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 58 parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/02ee6012-555a-41ad-a6a0-718d5998d923_isrForCrosstalkSources_860802_17.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/02ee6012-555a-41ad-a6a0-718d5998d923_isrForCrosstalkSources_860802_17.stderr parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 58 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 58 has memoization hash 8436774457f828bc74fe31bba189d9c2 parsl.dataflow.memoization INFO: Task 58 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa2020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 59 parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/71f4b674-34ab-42da-899e-1a7ea487efa0_isrForCrosstalkSources_860803_43.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/71f4b674-34ab-42da-899e-1a7ea487efa0_isrForCrosstalkSources_860803_43.stderr parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 59 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 59 has memoization hash 4c04b75d9ba013f5b27b2dbd40058359 parsl.dataflow.memoization INFO: Task 59 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa20c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 60 parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/4f4cf0c8-7664-4016-a60d-37a89711efcb_isrForCrosstalkSources_860803_58.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/4f4cf0c8-7664-4016-a60d-37a89711efcb_isrForCrosstalkSources_860803_58.stderr parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 60 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 60 has memoization hash a1275aff259cfcdbac0bb323f3364852 parsl.dataflow.memoization INFO: Task 60 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 61 parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/edad5722-c057-44b8-9e9e-82ec2dd36e9a_isrForCrosstalkSources_860803_32.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/edad5722-c057-44b8-9e9e-82ec2dd36e9a_isrForCrosstalkSources_860803_32.stderr parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 61 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 61 has memoization hash b59b6831a4f50fea196b074df2bbefb8 parsl.dataflow.memoization INFO: Task 61 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa22a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 62 parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/73250313-55ba-43dc-95f3-23d79a30b5a3_isrForCrosstalkSources_860802_36.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/73250313-55ba-43dc-95f3-23d79a30b5a3_isrForCrosstalkSources_860802_36.stderr parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 62 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 62 has memoization hash 986ea743e8646e55af009499ecd2df4e parsl.dataflow.memoization INFO: Task 62 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa2200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 63 parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/475c0ffc-b711-4acf-bab2-a17925ab3615_isrForCrosstalkSources_860803_53.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/475c0ffc-b711-4acf-bab2-a17925ab3615_isrForCrosstalkSources_860803_53.stderr parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 63 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 63 has memoization hash d1a469704153eae1f8a0e07d9641cec4 parsl.dataflow.memoization INFO: Task 63 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa2340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 64 parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/5210ac67-faa0-497f-ab66-3749279c9297_isrForCrosstalkSources_860803_55.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/5210ac67-faa0-497f-ab66-3749279c9297_isrForCrosstalkSources_860803_55.stderr parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 64 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 64 has memoization hash acbb09e2dfdb232abda8542a14255107 parsl.dataflow.memoization INFO: Task 64 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa23e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 65 parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/f9cab138-93a0-4750-a47a-925daf9eade6_isrForCrosstalkSources_860803_1.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/f9cab138-93a0-4750-a47a-925daf9eade6_isrForCrosstalkSources_860803_1.stderr parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 65 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 65 has memoization hash 80cf6ec34811760cc28525c76027a064 parsl.dataflow.memoization INFO: Task 65 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa2480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 65 try 0 launched on executor multi with executor id 66 parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/00d8ca68-90b2-4ef4-936e-3115ff835708_isrForCrosstalkSources_860802_51.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/00d8ca68-90b2-4ef4-936e-3115ff835708_isrForCrosstalkSources_860802_51.stderr parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 66 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 66 has memoization hash d52eb850cba7696e6abb18376b35e189 parsl.dataflow.memoization INFO: Task 66 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa25c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 67 parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/35d9dc53-d6d6-4ed6-a775-4ed9ed9a59f0_isrForCrosstalkSources_860802_38.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/35d9dc53-d6d6-4ed6-a775-4ed9ed9a59f0_isrForCrosstalkSources_860802_38.stderr parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 67 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 67 has memoization hash 35dbf53747cc5f1862d5a64e8b648731 parsl.dataflow.memoization INFO: Task 67 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa2520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 68 parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/540a11f8-6a8c-447c-ac52-8e6fbaf9a8ba_isrForCrosstalkSources_860802_39.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/540a11f8-6a8c-447c-ac52-8e6fbaf9a8ba_isrForCrosstalkSources_860802_39.stderr parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 68 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 68 has memoization hash 72db016853fcc9512cab1e1e8a3671ce parsl.dataflow.memoization INFO: Task 68 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 69 parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/e4d41443-9a67-4534-811a-0c2b1cdb36dc_isrForCrosstalkSources_860803_30.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/e4d41443-9a67-4534-811a-0c2b1cdb36dc_isrForCrosstalkSources_860803_30.stderr parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 69 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 69 has memoization hash ea5446c14026c25cff3bc1b6dbe380bc parsl.dataflow.memoization INFO: Task 69 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa2700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 70 parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/d36f88c2-844c-4ba5-a32f-3362e8c9d75a_isrForCrosstalkSources_860802_3.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/d36f88c2-844c-4ba5-a32f-3362e8c9d75a_isrForCrosstalkSources_860802_3.stderr parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 70 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 70 has memoization hash a41e451ea412ed4a67c800db8648dd7c parsl.dataflow.memoization INFO: Task 70 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 71 parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/f728410f-1518-4da8-b39b-8a820d5f3939_isrForCrosstalkSources_860801_60.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/f728410f-1518-4da8-b39b-8a820d5f3939_isrForCrosstalkSources_860801_60.stderr parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 71 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 71 has memoization hash 153ecd9d60b9742f205be51a0e3dc2b0 parsl.dataflow.memoization INFO: Task 71 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa28e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 71 try 0 launched on executor multi with executor id 72 parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/1abc9cab-40ab-458f-9335-1eb658c4023b_isrForCrosstalkSources_860802_7.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/1abc9cab-40ab-458f-9335-1eb658c4023b_isrForCrosstalkSources_860802_7.stderr parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 72 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 72 has memoization hash 0f7b0c8194e803995b122642b021cada parsl.dataflow.memoization INFO: Task 72 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa2840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 73 parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/ba94904a-2d1f-4ea3-99d8-ec9f12987d7a_isrForCrosstalkSources_860802_56.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/ba94904a-2d1f-4ea3-99d8-ec9f12987d7a_isrForCrosstalkSources_860802_56.stderr parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 73 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 73 has memoization hash 00f7d8f9266f92652827fb3f87ad880d parsl.dataflow.memoization INFO: Task 73 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 74 parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/4de187bc-91c9-41a7-b14e-6a7920522376_isrForCrosstalkSources_860803_35.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/4de187bc-91c9-41a7-b14e-6a7920522376_isrForCrosstalkSources_860803_35.stderr parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 74 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 74 has memoization hash 4e91bd8fc23a1132b39e109c681de14d parsl.dataflow.memoization INFO: Task 74 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa2a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 75 parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/e4ef14de-2d95-4904-8959-a07001dbb070_isrForCrosstalkSources_860802_44.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/e4ef14de-2d95-4904-8959-a07001dbb070_isrForCrosstalkSources_860802_44.stderr parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 75 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 75 has memoization hash 63de4c4edbc4eee5f547f829c30d0d9b parsl.dataflow.memoization INFO: Task 75 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa2ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 76 parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/976c5646-4ecf-45d4-8232-ca10ac21b804_isrForCrosstalkSources_860803_13.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/976c5646-4ecf-45d4-8232-ca10ac21b804_isrForCrosstalkSources_860803_13.stderr parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 76 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash dab9a89080f1521f97627e3dc0dc86ad parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa2c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 77 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/a591e198-d484-4b77-90ca-dd12e6fa44ea_isrForCrosstalkSources_860803_24.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/a591e198-d484-4b77-90ca-dd12e6fa44ea_isrForCrosstalkSources_860803_24.stderr parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 77 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 77 has memoization hash 9ef733eab4c57ad44e9b8e05c84c6273 parsl.dataflow.memoization INFO: Task 77 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 77 try 0 launched on executor multi with executor id 78 parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/25c7332f-73f0-4b0c-b164-cc85a3888a5f_isrForCrosstalkSources_860801_46.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/25c7332f-73f0-4b0c-b164-cc85a3888a5f_isrForCrosstalkSources_860801_46.stderr parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 78 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 78 has memoization hash 5175a34aa65edf5a3a459387e02c9cce parsl.dataflow.memoization INFO: Task 78 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa2ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 79 parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/747a3582-ba4c-4b69-b097-4e00084f8563_isrForCrosstalkSources_860801_24.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/747a3582-ba4c-4b69-b097-4e00084f8563_isrForCrosstalkSources_860801_24.stderr parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 79 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 79 has memoization hash d835a1b1fded08525b33efa35f7deea7 parsl.dataflow.memoization INFO: Task 79 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa2d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 80 parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/c9bf7867-3e01-4519-8f26-17774e809fd4_isrForCrosstalkSources_860803_23.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/c9bf7867-3e01-4519-8f26-17774e809fd4_isrForCrosstalkSources_860803_23.stderr parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 80 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 80 has memoization hash 860126842f66adbaa87ed4889061a242 parsl.dataflow.memoization INFO: Task 80 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 81 parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/2a1d8f9b-460c-4cb7-b552-ff9ef7e9450b_isrForCrosstalkSources_860802_29.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/2a1d8f9b-460c-4cb7-b552-ff9ef7e9450b_isrForCrosstalkSources_860802_29.stderr parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 81 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 81 has memoization hash 0408e81154f5d9a62d76bed298c6d4f0 parsl.dataflow.memoization INFO: Task 81 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 82 parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/809d64af-9464-4f4c-b0eb-c2f6b0fbf44f_isrForCrosstalkSources_860802_50.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/809d64af-9464-4f4c-b0eb-c2f6b0fbf44f_isrForCrosstalkSources_860802_50.stderr parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 82 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 82 has memoization hash 09950ff524e755c81f55c2a2489babf0 parsl.dataflow.memoization INFO: Task 82 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa2e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 83 parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/4d3f496e-a0a0-4aa8-a981-e67b8fcee380_isrForCrosstalkSources_860802_43.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/4d3f496e-a0a0-4aa8-a981-e67b8fcee380_isrForCrosstalkSources_860802_43.stderr parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 83 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 83 has memoization hash 779f74dff7aae9c7cdbd802dad4161a6 parsl.dataflow.memoization INFO: Task 83 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa2fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 83 try 0 launched on executor multi with executor id 84 parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/5110fffe-c6ef-4eb3-a658-aebb188e8f2a_isrForCrosstalkSources_860801_58.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/5110fffe-c6ef-4eb3-a658-aebb188e8f2a_isrForCrosstalkSources_860801_58.stderr parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 84 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 84 has memoization hash f1cfd3a8c2d792d3ba72271f9c6048c6 parsl.dataflow.memoization INFO: Task 84 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa3060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 85 parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/51cebf7f-e54d-4f04-8cc5-20b30f4c94a6_isrForCrosstalkSources_860802_8.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/51cebf7f-e54d-4f04-8cc5-20b30f4c94a6_isrForCrosstalkSources_860802_8.stderr parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 85 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 85 has memoization hash 6b361236fc075e53f2f458f0d0fac7f6 parsl.dataflow.memoization INFO: Task 85 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa3100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 86 parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/9ba5bd30-8427-442f-b459-848edd93673e_isrForCrosstalkSources_860803_20.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/9ba5bd30-8427-442f-b459-848edd93673e_isrForCrosstalkSources_860803_20.stderr parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 86 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 86 has memoization hash 49ffacff01996791cbe6555775da7271 parsl.dataflow.memoization INFO: Task 86 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa3240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 87 parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/f86023d0-0cd7-429a-aa7e-7052a3fe21cf_isrForCrosstalkSources_860801_52.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/f86023d0-0cd7-429a-aa7e-7052a3fe21cf_isrForCrosstalkSources_860801_52.stderr parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 87 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 87 has memoization hash a835c8b96d3c822a8a23a27d8da06bb4 parsl.dataflow.memoization INFO: Task 87 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa31a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 88 parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/00581f0c-8ec6-4398-bd41-2eea2db71020_isrForCrosstalkSources_860801_22.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/00581f0c-8ec6-4398-bd41-2eea2db71020_isrForCrosstalkSources_860801_22.stderr parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 88 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 88 has memoization hash 6a33605eb0278296d2c6a42956820d76 parsl.dataflow.memoization INFO: Task 88 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa32e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 89 parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/b5bdb31a-f78e-44a1-b450-a4d6f18e31ad_isrForCrosstalkSources_860803_3.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/b5bdb31a-f78e-44a1-b450-a4d6f18e31ad_isrForCrosstalkSources_860803_3.stderr parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 89 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 89 has memoization hash fe440ee5e00bc697c763cfe6b545cbbc parsl.dataflow.memoization INFO: Task 89 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa3380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 90 parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/7963aea7-cd1c-4258-aa4c-8e100a1f800c_isrForCrosstalkSources_860801_38.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/7963aea7-cd1c-4258-aa4c-8e100a1f800c_isrForCrosstalkSources_860801_38.stderr parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 90 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 90 has memoization hash c244e37d6e2e996626cf01d657abf113 parsl.dataflow.memoization INFO: Task 90 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 91 parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/87e135f3-b1b5-4365-836a-c0d7668bd9a6_isrForCrosstalkSources_860801_34.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/87e135f3-b1b5-4365-836a-c0d7668bd9a6_isrForCrosstalkSources_860801_34.stderr parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 91 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 91 has memoization hash 178f6aa571546be93d55cf135630d8ed parsl.dataflow.memoization INFO: Task 91 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa3560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 91 try 0 launched on executor multi with executor id 92 parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/a3a27500-75b6-47b3-b16e-1b189d55af2f_isrForCrosstalkSources_860802_14.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/a3a27500-75b6-47b3-b16e-1b189d55af2f_isrForCrosstalkSources_860802_14.stderr parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 92 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 92 has memoization hash 542bba6ae6449fc7d39a29955f3ef4c9 parsl.dataflow.memoization INFO: Task 92 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa34c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 93 parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/fc6b128a-0d4c-4b2a-b3f1-ca07ae121d63_isrForCrosstalkSources_860801_18.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/fc6b128a-0d4c-4b2a-b3f1-ca07ae121d63_isrForCrosstalkSources_860801_18.stderr parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 93 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 93 has memoization hash 77b15fcaa17574ad2bcdd0ee90c92b22 parsl.dataflow.memoization INFO: Task 93 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa3600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 94 parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/b828c7b7-5ca3-4385-a7ba-f55f0066b29d_isrForCrosstalkSources_860801_35.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/b828c7b7-5ca3-4385-a7ba-f55f0066b29d_isrForCrosstalkSources_860801_35.stderr parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 94 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 94 has memoization hash 57d5d8866391f8b76102844f48902a0b parsl.dataflow.memoization INFO: Task 94 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa36a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 95 parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/27a38fe1-9277-40f1-9596-30050bfe6161_isrForCrosstalkSources_860803_7.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/27a38fe1-9277-40f1-9596-30050bfe6161_isrForCrosstalkSources_860803_7.stderr parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 95 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 95 has memoization hash 4892b210c41fb6a9dba054845eed5193 parsl.dataflow.memoization INFO: Task 95 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa3740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 96 parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/f7694f75-90c6-4b88-82ad-17968c4c847a_isrForCrosstalkSources_860803_21.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/f7694f75-90c6-4b88-82ad-17968c4c847a_isrForCrosstalkSources_860803_21.stderr parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 96 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 96 has memoization hash 9a92012a12442f7c5056cdbaeddc07a1 parsl.dataflow.memoization INFO: Task 96 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa3880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 97 parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/a82613c5-0104-4dd4-91a2-b6f6551bd42e_isrForCrosstalkSources_860802_9.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/a82613c5-0104-4dd4-91a2-b6f6551bd42e_isrForCrosstalkSources_860802_9.stderr parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 97 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 97 has memoization hash 75101a036655cf881fd268b05034fdae parsl.dataflow.memoization INFO: Task 97 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa37e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 98 parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/5bcbe701-8194-45ed-b3eb-c09fba563b78_isrForCrosstalkSources_860802_19.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/5bcbe701-8194-45ed-b3eb-c09fba563b78_isrForCrosstalkSources_860802_19.stderr parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 98 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 98 has memoization hash 4f38f6c0038fb02e20585e6b813025b3 parsl.dataflow.memoization INFO: Task 98 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa3920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 99 parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/2517a6aa-3dce-47c0-aa9d-b6565c5494f0_isrForCrosstalkSources_860803_16.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/2517a6aa-3dce-47c0-aa9d-b6565c5494f0_isrForCrosstalkSources_860803_16.stderr parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 99 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 99 has memoization hash eb520e4e154a41730c9210e4d4fea901 parsl.dataflow.memoization INFO: Task 99 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa39c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 100 parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/9b0c12a2-b2b0-453d-be5b-9549d6e2baee_isrForCrosstalkSources_860803_6.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/9b0c12a2-b2b0-453d-be5b-9549d6e2baee_isrForCrosstalkSources_860803_6.stderr parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 100 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash a8c583f29c3fc7089f5225e4cb675297 parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa3a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 101 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/21359ac5-ac6b-43c2-aa01-cff16b1e12e6_isrForCrosstalkSources_860801_6.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/21359ac5-ac6b-43c2-aa01-cff16b1e12e6_isrForCrosstalkSources_860801_6.stderr parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 101 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 101 has memoization hash d0518c6d10fcc0430f6df3a133f21071 parsl.dataflow.memoization INFO: Task 101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa3ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-10 01:37:45 proc_lsst.multi:146 [INFO] found job 44401 in provider local proc_lsst.multi INFO: found job 44401 in provider local parsl.dataflow.dflow INFO: Parsl task 101 try 0 launched on executor multi with executor id 102 parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/ddc3b331-d93e-40a9-bcde-9594d7c6f587_isrForCrosstalkSources_860803_40.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/ddc3b331-d93e-40a9-bcde-9594d7c6f587_isrForCrosstalkSources_860803_40.stderr parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 102 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 102 has memoization hash a089add211b929bf5d36a2c77ac7e091 parsl.dataflow.memoization INFO: Task 102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f2fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 103 parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/49999342-acc3-483a-9d11-367d7688d326_isrForCrosstalkSources_860803_29.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/49999342-acc3-483a-9d11-367d7688d326_isrForCrosstalkSources_860803_29.stderr parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 103 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 103 has memoization hash 6ac35f8b106cfb3679c8f3847792315f parsl.dataflow.memoization INFO: Task 103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 104 parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/0519d253-9b16-4f97-9d1a-679632264654_isrForCrosstalkSources_860801_4.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/0519d253-9b16-4f97-9d1a-679632264654_isrForCrosstalkSources_860801_4.stderr parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 104 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 104 has memoization hash 572e468d95955f7e5ac52cff31c23cf3 parsl.dataflow.memoization INFO: Task 104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f3060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 105 parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/21f0cf9f-409e-45cb-bffa-3c2f694704b0_isrForCrosstalkSources_860802_4.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/21f0cf9f-409e-45cb-bffa-3c2f694704b0_isrForCrosstalkSources_860802_4.stderr parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 105 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 105 has memoization hash bcc00d3368f604f97b70a50c0bcea2c3 parsl.dataflow.memoization INFO: Task 105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 106 parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/69f54418-1cb6-4e1c-8aee-380b22c75a21_isrForCrosstalkSources_860802_59.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/69f54418-1cb6-4e1c-8aee-380b22c75a21_isrForCrosstalkSources_860802_59.stderr parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 106 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 106 has memoization hash 8989a44e0d7225152069b2499256fef8 parsl.dataflow.memoization INFO: Task 106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 107 parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/2f1c2e08-bb85-4b39-a3e3-63f8bb40ef54_isrForCrosstalkSources_860803_17.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/2f1c2e08-bb85-4b39-a3e3-63f8bb40ef54_isrForCrosstalkSources_860803_17.stderr parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 107 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 107 has memoization hash ef32ffbf71807e6a86f3c33154315ec2 parsl.dataflow.memoization INFO: Task 107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa3d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 108 parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/d7a95ccf-22a8-4534-854b-b0d9d8281752_isrForCrosstalkSources_860803_28.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/d7a95ccf-22a8-4534-854b-b0d9d8281752_isrForCrosstalkSources_860803_28.stderr parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 108 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 108 has memoization hash 0e726d5991d4f86c34ec619cf9a3113f parsl.dataflow.memoization INFO: Task 108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa3ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 109 parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/54e1b909-9b32-444b-ac0c-6d0fe2b0f0bc_isrForCrosstalkSources_860801_28.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/54e1b909-9b32-444b-ac0c-6d0fe2b0f0bc_isrForCrosstalkSources_860801_28.stderr parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 109 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 109 has memoization hash fa7920596d5e0a4d78f794553eea1467 parsl.dataflow.memoization INFO: Task 109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa3ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 110 parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/a01f995c-e6a8-4ea4-9822-4cbc18f0bde4_isrForCrosstalkSources_860801_9.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/a01f995c-e6a8-4ea4-9822-4cbc18f0bde4_isrForCrosstalkSources_860801_9.stderr parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 110 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 110 has memoization hash c10ee34403a681d27e78ad62492e2b83 parsl.dataflow.memoization INFO: Task 110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f0040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 111 parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/87e66c6c-c7c9-413a-9984-aed53b61173f_isrForCrosstalkSources_860803_57.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/87e66c6c-c7c9-413a-9984-aed53b61173f_isrForCrosstalkSources_860803_57.stderr parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 111 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 111 has memoization hash 3578c5bb198229b55878c13731b6323c parsl.dataflow.memoization INFO: Task 111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa3c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 111 parsl.dataflow.strategy DEBUG: Executor multi has 111 active tasks, 1/0 running/pending blocks, and 0 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 27 more blocks parsl.executors.status_handling INFO: Scaling out by 27 blocks parsl.executors.status_handling INFO: Allocated block ID 1 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1 2024-03-10 01:37:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 2024-03-10 01:37:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:37:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:37:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:37:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:37:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-10 01:37:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 112 parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/eb99815a-05ad-4a4c-a587-349ca1b97c98_isrForCrosstalkSources_860801_53.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/eb99815a-05ad-4a4c-a587-349ca1b97c98_isrForCrosstalkSources_860801_53.stderr parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 112 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 112 has memoization hash 9f37cc51b67dd5d9dec6d3a12cafbf22 parsl.dataflow.memoization INFO: Task 112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151105fa3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 113 parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/d179c47e-e6e7-415b-bcd5-e826fefc6629_isrForCrosstalkSources_860803_18.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/d179c47e-e6e7-415b-bcd5-e826fefc6629_isrForCrosstalkSources_860803_18.stderr parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 113 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 113 has memoization hash 747424205dda1dd6aa9a4925c87882f8 parsl.dataflow.memoization INFO: Task 113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f0220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 114 parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/bc385d76-403e-423e-ac51-aecafda90317_isrForCrosstalkSources_860801_8.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/bc385d76-403e-423e-ac51-aecafda90317_isrForCrosstalkSources_860801_8.stderr parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 114 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 114 has memoization hash cf55ac4e2b772de289bfb8cfd5aea4e2 parsl.dataflow.memoization INFO: Task 114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f2e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 115 parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/833a3654-d118-4003-b64a-5ce6de0b06ff_isrForCrosstalkSources_860802_25.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/833a3654-d118-4003-b64a-5ce6de0b06ff_isrForCrosstalkSources_860802_25.stderr parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 115 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 115 has memoization hash dbbe1b83a6f06a26402a9a3c75d8f1bf parsl.dataflow.memoization INFO: Task 115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f0400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 116 parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/e0ff9c89-b24c-4c7e-9332-6dc407000d12_isrForCrosstalkSources_860803_60.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/e0ff9c89-b24c-4c7e-9332-6dc407000d12_isrForCrosstalkSources_860803_60.stderr parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 116 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 116 has memoization hash d4cf5b2b26ed3a3696a3649054cfd4f1 parsl.dataflow.memoization INFO: Task 116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f02c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 117 parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/61332636-d9c2-482c-b203-fd7ec6c37fc2_isrForCrosstalkSources_860802_10.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/61332636-d9c2-482c-b203-fd7ec6c37fc2_isrForCrosstalkSources_860802_10.stderr parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 117 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 117 has memoization hash 2f3874d712b07f06379cf522d600a3f5 parsl.dataflow.memoization INFO: Task 117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f04a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 118 parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/15be0586-0605-4322-bc0c-facc3682322c_isrForCrosstalkSources_860802_24.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/15be0586-0605-4322-bc0c-facc3682322c_isrForCrosstalkSources_860802_24.stderr parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 118 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash 129dcfa7b50cf1bd68593dbd6f8351a3 parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f0360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 119 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/cc4ca081-9b18-417f-b625-09f695f9d91b_isrForCrosstalkSources_860801_40.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/cc4ca081-9b18-417f-b625-09f695f9d91b_isrForCrosstalkSources_860801_40.stderr parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 119 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 119 has memoization hash 0dabe1770d40dd930fec3f387118009d parsl.dataflow.memoization INFO: Task 119 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f00e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 119 try 0 launched on executor multi with executor id 120 parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/166eb073-2976-422b-8d8d-fc13b973b3e9_isrForCrosstalkSources_860803_62.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/166eb073-2976-422b-8d8d-fc13b973b3e9_isrForCrosstalkSources_860803_62.stderr parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 120 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 120 has memoization hash 2691a799c7a544599c32eb2333c799d2 parsl.dataflow.memoization INFO: Task 120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f3ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 121 parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/0856ee1c-1318-4375-a763-820f2423dd9e_isrForCrosstalkSources_860803_2.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/0856ee1c-1318-4375-a763-820f2423dd9e_isrForCrosstalkSources_860803_2.stderr parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 121 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 121 has memoization hash 49faa0b102e2b5222d28348353acea8d parsl.dataflow.memoization INFO: Task 121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f0680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 122 parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/804d7d39-346a-4b15-a35b-501c9b1b7881_isrForCrosstalkSources_860802_49.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/804d7d39-346a-4b15-a35b-501c9b1b7881_isrForCrosstalkSources_860802_49.stderr parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 122 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 122 has memoization hash 0b2f5975e0f8a4cc3438e72d1fb53691 parsl.dataflow.memoization INFO: Task 122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 123 parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/e3dab589-8988-404b-8386-08b9c179e8bb_isrForCrosstalkSources_860801_50.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/e3dab589-8988-404b-8386-08b9c179e8bb_isrForCrosstalkSources_860801_50.stderr parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 123 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 123 has memoization hash 932ded8fa6fc8a87775df2c94686b80b parsl.dataflow.memoization INFO: Task 123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f07c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 124 parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/a6fa9773-9d34-4105-8986-06c342035102_isrForCrosstalkSources_860803_37.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/a6fa9773-9d34-4105-8986-06c342035102_isrForCrosstalkSources_860803_37.stderr parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 124 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 124 has memoization hash 980c7a8c88be4f1a3b1650343078a406 parsl.dataflow.memoization INFO: Task 124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f3ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 125 parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/bd47f21a-1814-480a-98c4-5ab5b2011ad5_isrForCrosstalkSources_860803_59.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/bd47f21a-1814-480a-98c4-5ab5b2011ad5_isrForCrosstalkSources_860803_59.stderr parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 125 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 125 has memoization hash 8bae0e849bb7c4b21d14f22648f43b18 parsl.dataflow.memoization INFO: Task 125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f3c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 125 try 0 launched on executor multi with executor id 126 parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/59a7407b-5004-4144-9b0f-72f09780ed21_isrForCrosstalkSources_860803_41.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/59a7407b-5004-4144-9b0f-72f09780ed21_isrForCrosstalkSources_860803_41.stderr parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 126 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 126 has memoization hash 065ef561e89805827a0477062b8a051b parsl.dataflow.memoization INFO: Task 126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f0860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 127 parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/77663be1-bd68-4e79-85d4-7299414fff75_isrForCrosstalkSources_860801_48.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/77663be1-bd68-4e79-85d4-7299414fff75_isrForCrosstalkSources_860801_48.stderr parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 127 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 127 has memoization hash 950d54daafe597a22228ebc5b5396bf6 parsl.dataflow.memoization INFO: Task 127 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f0900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 127 try 0 launched on executor multi with executor id 128 parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/98d87c94-f6c0-4adb-b155-b3896b186e61_isrForCrosstalkSources_860801_10.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/98d87c94-f6c0-4adb-b155-b3896b186e61_isrForCrosstalkSources_860801_10.stderr parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 128 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 128 has memoization hash 995d997c2fb469b4656b8576f25b9f8b parsl.dataflow.memoization INFO: Task 128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f09a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 129 parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/de11ad99-eff4-4436-a9c7-10b0c253e178_isrForCrosstalkSources_860802_22.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/de11ad99-eff4-4436-a9c7-10b0c253e178_isrForCrosstalkSources_860802_22.stderr parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 129 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 129 has memoization hash 9ea14aeaf9a78d44153682535569cc3b parsl.dataflow.memoization INFO: Task 129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f0ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 129 try 0 launched on executor multi with executor id 130 parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/c563faf9-eee6-4cb8-bc60-7141ae803d72_isrForCrosstalkSources_860803_10.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/c563faf9-eee6-4cb8-bc60-7141ae803d72_isrForCrosstalkSources_860803_10.stderr parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 130 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 130 has memoization hash aed4684df6b82653694e7a34dbcb53c8 parsl.dataflow.memoization INFO: Task 130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 131 parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/dc885483-6018-4b01-b102-59b7e2d7f926_isrForCrosstalkSources_860802_30.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/dc885483-6018-4b01-b102-59b7e2d7f926_isrForCrosstalkSources_860802_30.stderr parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 131 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 131 has memoization hash 630ac7f260f710de82d8e25d29b27d97 parsl.dataflow.memoization INFO: Task 131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f0c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 131 try 0 launched on executor multi with executor id 132 parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/7160d22e-bfb8-4ba4-a69c-1d08faba322b_isrForCrosstalkSources_860801_59.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/7160d22e-bfb8-4ba4-a69c-1d08faba322b_isrForCrosstalkSources_860801_59.stderr parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 132 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 132 has memoization hash fa3db82dc4eb76b61ec597f2a5e038e0 parsl.dataflow.memoization INFO: Task 132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f0720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 133 parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/d0f9db30-118c-40cc-af8a-ec84f1efbad3_isrForCrosstalkSources_860802_53.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/d0f9db30-118c-40cc-af8a-ec84f1efbad3_isrForCrosstalkSources_860802_53.stderr parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 133 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 133 has memoization hash baadb6b43e9769a6e169464c6ed6980a parsl.dataflow.memoization INFO: Task 133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f0a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 133 try 0 launched on executor multi with executor id 134 parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/71a1ae0d-7c1f-4013-aa88-b18aadba12b9_isrForCrosstalkSources_860803_33.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/71a1ae0d-7c1f-4013-aa88-b18aadba12b9_isrForCrosstalkSources_860803_33.stderr parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 134 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 134 has memoization hash 3b83311e6ba24f3e3043274506f16b06 parsl.dataflow.memoization INFO: Task 134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f0d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 135 parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/9f6071e7-22c4-4252-aabe-a8f738cc7abb_isrForCrosstalkSources_860801_32.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/9f6071e7-22c4-4252-aabe-a8f738cc7abb_isrForCrosstalkSources_860801_32.stderr parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 135 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 135 has memoization hash c2b0d41a74a9296e99c3afa45b4abe95 parsl.dataflow.memoization INFO: Task 135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f0e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 135 try 0 launched on executor multi with executor id 136 parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/83a3c6a4-493c-4021-b89b-af981dbe904c_isrForCrosstalkSources_860802_5.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/83a3c6a4-493c-4021-b89b-af981dbe904c_isrForCrosstalkSources_860802_5.stderr parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 136 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 136 has memoization hash b0b25036656cedd27aac0d1be281f79e parsl.dataflow.memoization INFO: Task 136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f0cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 137 parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/126de834-1baf-4fcc-ae27-6a4baebd8911_isrForCrosstalkSources_860803_25.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/126de834-1baf-4fcc-ae27-6a4baebd8911_isrForCrosstalkSources_860803_25.stderr parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 137 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 137 has memoization hash c4ff91a6ae55e487e0ff8ffd692cadaa parsl.dataflow.memoization INFO: Task 137 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f0f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 137 try 0 launched on executor multi with executor id 138 parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/4326038d-5407-4345-a3c8-ea7aeceebd3d_isrForCrosstalkSources_860801_14.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/4326038d-5407-4345-a3c8-ea7aeceebd3d_isrForCrosstalkSources_860801_14.stderr parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 138 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 138 has memoization hash aa1f11f61089ee79319c49bc9be7d3c0 parsl.dataflow.memoization INFO: Task 138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-10 01:37:45 proc_lsst.multi:170 [INFO] [multi] job_id 16971842 proc_lsst.multi INFO: [multi] job_id 16971842 2024-03-10 01:37:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:37:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:45 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16971842 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16971842 parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 16971842 parsl.executors.status_handling INFO: Allocated block ID 2 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2 2024-03-10 01:37:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 2024-03-10 01:37:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:37:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:37:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:37:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:37:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:37:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 139 parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/722dd519-c143-475b-a818-fd7d0407d886_isrForCrosstalkSources_860803_46.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/722dd519-c143-475b-a818-fd7d0407d886_isrForCrosstalkSources_860803_46.stderr parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 139 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 139 has memoization hash 01af022b7a11ed81e84476affa933691 parsl.dataflow.memoization INFO: Task 139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 139 try 0 launched on executor multi with executor id 140 parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/beb6495b-922f-42af-87ab-249cc3bb3fd5_isrForCrosstalkSources_860802_54.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/beb6495b-922f-42af-87ab-249cc3bb3fd5_isrForCrosstalkSources_860802_54.stderr parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 140 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 140 has memoization hash 8ecbb98d85a7c6036ca72e39d63772a4 parsl.dataflow.memoization INFO: Task 140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f0fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 141 parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/994650d7-3232-4985-8b3c-b3b001b50931_isrForCrosstalkSources_860801_29.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/994650d7-3232-4985-8b3c-b3b001b50931_isrForCrosstalkSources_860801_29.stderr parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 141 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 141 has memoization hash b5b4c015c02befccc8357b51a580636a parsl.dataflow.memoization INFO: Task 141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f22a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 141 try 0 launched on executor multi with executor id 142 parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/59eca0b7-5253-4fc8-a4b1-77acbc256128_isrForCrosstalkSources_860801_2.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/59eca0b7-5253-4fc8-a4b1-77acbc256128_isrForCrosstalkSources_860801_2.stderr parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 142 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 142 has memoization hash 4132e504bc1a88ac31cfc3a66b6023c9 parsl.dataflow.memoization INFO: Task 142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f13a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 143 parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/181a830c-9ef4-4a52-bb78-2b106c9c08e9_isrForCrosstalkSources_860801_55.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/181a830c-9ef4-4a52-bb78-2b106c9c08e9_isrForCrosstalkSources_860801_55.stderr parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 143 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 143 has memoization hash a6fbb106b19e1fed70d377b6600acafb parsl.dataflow.memoization INFO: Task 143 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f23e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 143 try 0 launched on executor multi with executor id 144 parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/e89327c2-d9f7-4a4b-b26f-3fb2ea0b08ef_isrForCrosstalkSources_860802_57.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/e89327c2-d9f7-4a4b-b26f-3fb2ea0b08ef_isrForCrosstalkSources_860802_57.stderr parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 144 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 144 has memoization hash 79f85800406f6ce1d6e0c4f9789d2d0e parsl.dataflow.memoization INFO: Task 144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f1300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 145 parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/00c5577e-0078-4f29-8a5f-6777901d9c3d_isrForCrosstalkSources_860802_48.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/00c5577e-0078-4f29-8a5f-6777901d9c3d_isrForCrosstalkSources_860802_48.stderr parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 145 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 145 has memoization hash 723e56fbcb99b092576eb8a5b88a68f9 parsl.dataflow.memoization INFO: Task 145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 145 try 0 launched on executor multi with executor id 146 parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/c74c5d3c-a990-4a76-9e2c-b7a3d8d957a8_isrForCrosstalkSources_860803_51.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/c74c5d3c-a990-4a76-9e2c-b7a3d8d957a8_isrForCrosstalkSources_860803_51.stderr parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 146 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 146 has memoization hash 5e8fe896d2d8d533124bf128b4213504 parsl.dataflow.memoization INFO: Task 146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f1440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 146 try 0 launched on executor multi with executor id 147 parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/7f3f7278-121d-41e5-881a-038541fec844_isrForCrosstalkSources_860802_41.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/7f3f7278-121d-41e5-881a-038541fec844_isrForCrosstalkSources_860802_41.stderr parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 147 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 147 has memoization hash 4426568f1d102c4fa6d9330c3acc7285 parsl.dataflow.memoization INFO: Task 147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f1580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 147 try 0 launched on executor multi with executor id 148 parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/9c0c085e-dcc4-4e1b-820c-dce455d7ae8e_isrForCrosstalkSources_860802_2.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/9c0c085e-dcc4-4e1b-820c-dce455d7ae8e_isrForCrosstalkSources_860802_2.stderr parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 148 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 148 has memoization hash eb5901ed24b0ba17c180dcff23b30c41 parsl.dataflow.memoization INFO: Task 148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f1120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 149 parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/0898a38a-def6-476a-ad7e-828b92edaa39_isrForCrosstalkSources_860801_30.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/0898a38a-def6-476a-ad7e-828b92edaa39_isrForCrosstalkSources_860801_30.stderr parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 149 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 149 has memoization hash 8a1ce2fea28dbf63ad913ad4f5f7afd0 parsl.dataflow.memoization INFO: Task 149 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 149 try 0 launched on executor multi with executor id 150 parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/b9eb3d40-df89-4329-89c3-fa1bd6dc1419_isrForCrosstalkSources_860803_19.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/b9eb3d40-df89-4329-89c3-fa1bd6dc1419_isrForCrosstalkSources_860803_19.stderr parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 150 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 150 has memoization hash 121366f266bc31c1f1418033d5f12dd5 parsl.dataflow.memoization INFO: Task 150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f14e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 151 parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/7d5ebc1c-c542-403c-9344-7d0356b96790_isrForCrosstalkSources_860803_39.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/7d5ebc1c-c542-403c-9344-7d0356b96790_isrForCrosstalkSources_860803_39.stderr parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 151 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 151 has memoization hash 3fac9f03fd0d0bcfb16c9e7e5b23b588 parsl.dataflow.memoization INFO: Task 151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f1620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 151 try 0 launched on executor multi with executor id 152 parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/c6427985-d3f6-4646-bdf8-fe311d92eb29_isrForCrosstalkSources_860802_31.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/c6427985-d3f6-4646-bdf8-fe311d92eb29_isrForCrosstalkSources_860802_31.stderr parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 152 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 152 has memoization hash 0ccaaf4a52a77da76ec9867c4142d281 parsl.dataflow.memoization INFO: Task 152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 152 try 0 launched on executor multi with executor id 153 parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/6e0817a2-fc92-4f03-9f83-f1794a05a75e_isrForCrosstalkSources_860801_20.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/6e0817a2-fc92-4f03-9f83-f1794a05a75e_isrForCrosstalkSources_860801_20.stderr parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 153 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 153 has memoization hash 1cff7dcda069cb91130bd4a66b8fe949 parsl.dataflow.memoization INFO: Task 153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 153 try 0 launched on executor multi with executor id 154 parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/95cc8e5e-97dc-4d0b-b91f-16fffa5f1116_isrForCrosstalkSources_860802_27.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/95cc8e5e-97dc-4d0b-b91f-16fffa5f1116_isrForCrosstalkSources_860802_27.stderr parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 154 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 154 has memoization hash c898aef8f19809b04e7953e32020f143 parsl.dataflow.memoization INFO: Task 154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f1760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 155 parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/18b52ba3-4ed7-435d-87d3-3367e2d098a8_isrForCrosstalkSources_860802_34.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/18b52ba3-4ed7-435d-87d3-3367e2d098a8_isrForCrosstalkSources_860802_34.stderr parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 155 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 155 has memoization hash 7f5ada2967edee107ebf05db61be56b0 parsl.dataflow.memoization INFO: Task 155 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f1940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 155 try 0 launched on executor multi with executor id 156 parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/6f51b79f-6c0b-46c2-a3de-f58bb74bef6a_isrForCrosstalkSources_860801_33.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/6f51b79f-6c0b-46c2-a3de-f58bb74bef6a_isrForCrosstalkSources_860801_33.stderr parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 156 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 156 has memoization hash 2740280555ece9376a046b56f106e3fa parsl.dataflow.memoization INFO: Task 156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f1a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 157 parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/50d61ceb-9e4a-47db-ad01-d00a6f2288b2_isrForCrosstalkSources_860802_45.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/50d61ceb-9e4a-47db-ad01-d00a6f2288b2_isrForCrosstalkSources_860802_45.stderr parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 157 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 157 has memoization hash d1f8ebbd5b9e9f0f19fb8ff1b94f9f4a parsl.dataflow.memoization INFO: Task 157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f18a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 157 try 0 launched on executor multi with executor id 158 parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/0172fc64-d898-46c9-bbd0-32c59b7d415a_isrForCrosstalkSources_860802_52.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/0172fc64-d898-46c9-bbd0-32c59b7d415a_isrForCrosstalkSources_860802_52.stderr parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 158 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 158 has memoization hash a19837748da61df3a481a786580de24e parsl.dataflow.memoization INFO: Task 158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 158 try 0 launched on executor multi with executor id 159 parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/a7f3ca5b-e36d-4b7d-9525-bc1057269598_isrForCrosstalkSources_860801_36.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/a7f3ca5b-e36d-4b7d-9525-bc1057269598_isrForCrosstalkSources_860801_36.stderr parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 159 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 159 has memoization hash f00e532dd159e4489e66158cb1658239 parsl.dataflow.memoization INFO: Task 159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f1bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 159 try 0 launched on executor multi with executor id 160 parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/3e0cfb2c-a231-4a12-a2dc-81cb26f7970e_isrForCrosstalkSources_860801_16.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/3e0cfb2c-a231-4a12-a2dc-81cb26f7970e_isrForCrosstalkSources_860801_16.stderr 2024-03-10 01:37:46 proc_lsst.multi:170 [INFO] [multi] job_id 16971843 proc_lsst.multi INFO: [multi] job_id 16971843 2024-03-10 01:37:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-10 01:37:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:46 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16971843 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16971843 parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 16971843 parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.executors.status_handling INFO: Allocated block ID 3 parsl.dataflow.dflow INFO: Task 160 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3 2024-03-10 01:37:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 2024-03-10 01:37:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:37:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:37:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:37:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:37:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-10 01:37:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 160 has memoization hash ba907e883caf7c6152232aaba0cf6585 parsl.dataflow.memoization INFO: Task 160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f2d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 160 try 0 launched on executor multi with executor id 161 parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/399547a3-5e9b-45f5-ab86-24246476e7f9_isrForCrosstalkSources_860801_47.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/399547a3-5e9b-45f5-ab86-24246476e7f9_isrForCrosstalkSources_860801_47.stderr parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 161 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 161 has memoization hash 7f998a9515d646a0fb561980668e9c3a parsl.dataflow.memoization INFO: Task 161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 161 try 0 launched on executor multi with executor id 162 parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/beac8950-e65f-49ac-9036-833a11e1722b_isrForCrosstalkSources_860802_32.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/beac8950-e65f-49ac-9036-833a11e1722b_isrForCrosstalkSources_860802_32.stderr parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 162 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 162 has memoization hash f698cbc222a732e195557216910bd72d parsl.dataflow.memoization INFO: Task 162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f05e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 163 parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/205957c0-a6dc-4b69-b788-51bce0808c6a_isrForCrosstalkSources_860802_23.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/205957c0-a6dc-4b69-b788-51bce0808c6a_isrForCrosstalkSources_860802_23.stderr parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 163 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 163 has memoization hash 0ef1dfdf0d199a093b594d6d5b29b005 parsl.dataflow.memoization INFO: Task 163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f2ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 163 try 0 launched on executor multi with executor id 164 parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/61110114-d436-48ea-b9bc-97c5c25c5a53_isrForCrosstalkSources_860802_13.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/61110114-d436-48ea-b9bc-97c5c25c5a53_isrForCrosstalkSources_860802_13.stderr parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 164 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 164 has memoization hash 8d4ff401e15d6e2cc1195b1dffa33895 parsl.dataflow.memoization INFO: Task 164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f2c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 165 parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/84aa977f-c390-49bc-9c63-9737468315be_isrForCrosstalkSources_860803_49.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/84aa977f-c390-49bc-9c63-9737468315be_isrForCrosstalkSources_860803_49.stderr parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 165 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 165 has memoization hash ef12ecbe7860d24b73cd3a3ba4d17b03 parsl.dataflow.memoization INFO: Task 165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 165 try 0 launched on executor multi with executor id 166 parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/cf9777a9-a65f-4bff-a8f1-e87f96dc1796_isrForCrosstalkSources_860801_61.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/cf9777a9-a65f-4bff-a8f1-e87f96dc1796_isrForCrosstalkSources_860801_61.stderr parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 166 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 166 has memoization hash 042d1a61aae358d291aba098cfb622b0 parsl.dataflow.memoization INFO: Task 166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f28e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 167 parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/543ed78e-36ea-495f-98d0-019f3b6df0b6_isrForCrosstalkSources_860803_11.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/543ed78e-36ea-495f-98d0-019f3b6df0b6_isrForCrosstalkSources_860803_11.stderr parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 167 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 167 has memoization hash 0413362d7e294575446d83cf24977417 parsl.dataflow.memoization INFO: Task 167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f2840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 167 try 0 launched on executor multi with executor id 168 parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/50950038-3566-4675-8af9-c8a45660883b_isrForCrosstalkSources_860802_15.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/50950038-3566-4675-8af9-c8a45660883b_isrForCrosstalkSources_860802_15.stderr parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 168 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 168 has memoization hash 199df6efa6f80155d45dfc600f505a4d parsl.dataflow.memoization INFO: Task 168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f2a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 169 parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/74253bf0-ffc2-44f1-806d-3fb0e3332a21_isrForCrosstalkSources_860801_12.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/74253bf0-ffc2-44f1-806d-3fb0e3332a21_isrForCrosstalkSources_860801_12.stderr parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 169 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 169 has memoization hash ba95052539a23860b978056a96d2d432 parsl.dataflow.memoization INFO: Task 169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 169 try 0 launched on executor multi with executor id 170 parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/1cecc060-209e-4469-8abd-2c5a88865479_isrForCrosstalkSources_860802_20.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/1cecc060-209e-4469-8abd-2c5a88865479_isrForCrosstalkSources_860802_20.stderr parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 170 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 170 has memoization hash f7458c977ee91028adea10bc5bb2778f parsl.dataflow.memoization INFO: Task 170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 170 try 0 launched on executor multi with executor id 171 parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/55d81396-0c3c-41b6-a16f-87f84ce4e007_isrForCrosstalkSources_860803_38.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/55d81396-0c3c-41b6-a16f-87f84ce4e007_isrForCrosstalkSources_860803_38.stderr parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 171 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 171 has memoization hash e6139d722d6bb6314f9b78d1776a6151 parsl.dataflow.memoization INFO: Task 171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 171 try 0 launched on executor multi with executor id 172 parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/274e5d19-82cb-4829-b992-9c16857efe7a_isrForCrosstalkSources_860801_15.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/274e5d19-82cb-4829-b992-9c16857efe7a_isrForCrosstalkSources_860801_15.stderr parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 172 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 172 has memoization hash af0bc02e0f12d1cc99788f413908d8f3 parsl.dataflow.memoization INFO: Task 172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f2700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 173 parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/49260c23-be57-46fa-aa07-eddbd804d1f1_isrForCrosstalkSources_860802_40.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/49260c23-be57-46fa-aa07-eddbd804d1f1_isrForCrosstalkSources_860802_40.stderr parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 173 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 173 has memoization hash 38a0a75566e6d42170c61dd3413e6c40 parsl.dataflow.memoization INFO: Task 173 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f2520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 173 try 0 launched on executor multi with executor id 174 parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/74bcbe5d-da66-4521-b66a-1abc02317ab5_isrForCrosstalkSources_860801_57.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/74bcbe5d-da66-4521-b66a-1abc02317ab5_isrForCrosstalkSources_860801_57.stderr parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 174 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 174 has memoization hash e1b276787a563f92e30e3d4e45f296b7 parsl.dataflow.memoization INFO: Task 174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f2340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 175 parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/54203813-1fd9-4a12-86c1-e5a583dd1337_isrForCrosstalkSources_860801_39.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/54203813-1fd9-4a12-86c1-e5a583dd1337_isrForCrosstalkSources_860801_39.stderr parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 175 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 175 has memoization hash 942875531bf4b6f47ae518556769fc36 parsl.dataflow.memoization INFO: Task 175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f25c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 175 try 0 launched on executor multi with executor id 176 parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/c3736c01-59ac-4475-9770-f4239d498e04_isrForCrosstalkSources_860801_21.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/c3736c01-59ac-4475-9770-f4239d498e04_isrForCrosstalkSources_860801_21.stderr parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 176 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 176 has memoization hash 877440132e04c21ef4981a252f9c0055 parsl.dataflow.memoization INFO: Task 176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f3b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 177 parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/3a93d494-8a79-4ea0-97b4-2551c7bf94d2_isrForCrosstalkSources_860802_6.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/3a93d494-8a79-4ea0-97b4-2551c7bf94d2_isrForCrosstalkSources_860802_6.stderr parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 177 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 177 has memoization hash 3aedb4a233a80f1ae7b3c14aebaa89bc parsl.dataflow.memoization INFO: Task 177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f2480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 177 try 0 launched on executor multi with executor id 178 parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/19bb75c2-8ac0-42c3-ada0-d8901d8715c4_isrForCrosstalkSources_860802_16.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/19bb75c2-8ac0-42c3-ada0-d8901d8715c4_isrForCrosstalkSources_860802_16.stderr parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 178 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 178 has memoization hash c4d9a7449d7da6dca8f3fbfc17b3c5db parsl.dataflow.memoization INFO: Task 178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f3ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 179 parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/3857ff2a-600b-4a75-8908-58f6f129342f_isrForCrosstalkSources_860803_26.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/3857ff2a-600b-4a75-8908-58f6f129342f_isrForCrosstalkSources_860803_26.stderr parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 179 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 179 has memoization hash 2af0f58df387d85a9d1782ff00919732 parsl.dataflow.memoization INFO: Task 179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f3920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 179 try 0 launched on executor multi with executor id 180 parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/f945a938-0032-4edd-a0cf-50e35a92acc1_isrForCrosstalkSources_860803_36.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/f945a938-0032-4edd-a0cf-50e35a92acc1_isrForCrosstalkSources_860803_36.stderr parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 180 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 180 has memoization hash 5c1a55ab9784d0e66cd1ec684e6a0077 parsl.dataflow.memoization INFO: Task 180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f3a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-10 01:37:46 proc_lsst.multi:170 [INFO] [multi] job_id 16971844 proc_lsst.multi INFO: [multi] job_id 16971844 2024-03-10 01:37:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-10 01:37:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:46 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16971844 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16971844 parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 16971844 parsl.executors.status_handling INFO: Allocated block ID 4 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4 2024-03-10 01:37:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 2024-03-10 01:37:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:37:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:37:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:37:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:37:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-10 01:37:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 181 parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/fcc57517-6c72-4b38-85cc-1a0f544e47ea_isrForCrosstalkSources_860803_61.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/fcc57517-6c72-4b38-85cc-1a0f544e47ea_isrForCrosstalkSources_860803_61.stderr parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 181 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 181 has memoization hash c8f56d1271afe455c8220c2a7f61246b parsl.dataflow.memoization INFO: Task 181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f39c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 181 try 0 launched on executor multi with executor id 182 parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/a6062270-47cd-4cdf-a3dd-607309d5d4d0_isrForCrosstalkSources_860801_37.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/a6062270-47cd-4cdf-a3dd-607309d5d4d0_isrForCrosstalkSources_860801_37.stderr parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 182 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 182 has memoization hash 8e47f1ea806d8e4c67db22b76e29c323 parsl.dataflow.memoization INFO: Task 182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f37e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 183 parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/72711c40-2aab-4425-a368-9e30ed09c2ae_isrForCrosstalkSources_860803_50.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/72711c40-2aab-4425-a368-9e30ed09c2ae_isrForCrosstalkSources_860803_50.stderr parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 183 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 183 has memoization hash 0c4a16e56d9898c64278b0597dc4863c parsl.dataflow.memoization INFO: Task 183 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f3740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 183 try 0 launched on executor multi with executor id 184 parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/9176a5d3-372d-4f4e-9f34-ead222248b62_isrForCrosstalkSources_860802_46.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/9176a5d3-372d-4f4e-9f34-ead222248b62_isrForCrosstalkSources_860802_46.stderr parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 184 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 184 has memoization hash 9d480f47966a2111277df580619e5070 parsl.dataflow.memoization INFO: Task 184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f3880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 185 parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/fc96a945-5135-42b3-8f24-3a0ae62d6e21_isrForCrosstalkSources_860801_51.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/fc96a945-5135-42b3-8f24-3a0ae62d6e21_isrForCrosstalkSources_860801_51.stderr parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 185 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 185 has memoization hash c03ffb20fb27c157f278fc82ee143616 parsl.dataflow.memoization INFO: Task 185 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1511049f0180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 185 try 0 launched on executor multi with executor id 186 parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/86411d0f-3d53-41a2-be13-e0a6e503dcc7_isrForCrosstalkSources_860803_31.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/86411d0f-3d53-41a2-be13-e0a6e503dcc7_isrForCrosstalkSources_860803_31.stderr 2024-03-10 01:37:47 proc_lsst.multi:170 [INFO] [multi] job_id 16971845 proc_lsst.multi INFO: [multi] job_id 16971845 2024-03-10 01:37:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-10 01:37:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:47 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16971845 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16971845 parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 16971845 parsl.executors.status_handling INFO: Allocated block ID 5 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5 2024-03-10 01:37:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 2024-03-10 01:37:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:37:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:37:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:37:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:37:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-10 01:37:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 01:37:47 proc_lsst.multi:170 [INFO] [multi] job_id 16971846 proc_lsst.multi INFO: [multi] job_id 16971846 2024-03-10 01:37:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-10 01:37:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:47 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16971846 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16971846 parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 16971846 parsl.executors.status_handling INFO: Allocated block ID 6 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6 2024-03-10 01:37:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 2024-03-10 01:37:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:37:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:37:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:37:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:37:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-10 01:37:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 01:37:47 proc_lsst.multi:170 [INFO] [multi] job_id 16971847 proc_lsst.multi INFO: [multi] job_id 16971847 2024-03-10 01:37:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-10 01:37:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:47 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16971847 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16971847 parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 16971847 parsl.executors.status_handling INFO: Allocated block ID 7 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7 2024-03-10 01:37:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 2024-03-10 01:37:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:37:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:37:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:37:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:37:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-10 01:37:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 01:37:48 proc_lsst.multi:170 [INFO] [multi] job_id 16971848 proc_lsst.multi INFO: [multi] job_id 16971848 2024-03-10 01:37:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-10 01:37:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:48 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16971848 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16971848 parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 16971848 parsl.executors.status_handling INFO: Allocated block ID 8 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8 2024-03-10 01:37:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 2024-03-10 01:37:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:37:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:37:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:37:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:37:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-10 01:37:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 01:37:48 proc_lsst.multi:170 [INFO] [multi] job_id 16971849 proc_lsst.multi INFO: [multi] job_id 16971849 2024-03-10 01:37:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-10 01:37:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:48 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16971849 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16971849 parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 16971849 parsl.executors.status_handling INFO: Allocated block ID 9 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9 2024-03-10 01:37:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 2024-03-10 01:37:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:37:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:37:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:37:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:37:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-10 01:37:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 01:37:49 proc_lsst.multi:170 [INFO] [multi] job_id 16971850 proc_lsst.multi INFO: [multi] job_id 16971850 2024-03-10 01:37:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-10 01:37:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:49 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16971850 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16971850 parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 16971850 parsl.executors.status_handling INFO: Allocated block ID 10 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10 2024-03-10 01:37:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 2024-03-10 01:37:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:37:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:37:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:37:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:37:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-10 01:37:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 01:37:49 proc_lsst.multi:170 [INFO] [multi] job_id 16971851 proc_lsst.multi INFO: [multi] job_id 16971851 2024-03-10 01:37:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-10 01:37:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:49 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16971851 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16971851 parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 16971851 parsl.executors.status_handling INFO: Allocated block ID 11 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11 2024-03-10 01:37:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 2024-03-10 01:37:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:37:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:37:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:37:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:37:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-10 01:37:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 01:37:50 proc_lsst.multi:170 [INFO] [multi] job_id 16971852 proc_lsst.multi INFO: [multi] job_id 16971852 2024-03-10 01:37:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-10 01:37:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:50 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16971852 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16971852 parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 16971852 parsl.executors.status_handling INFO: Allocated block ID 12 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12 2024-03-10 01:37:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 2024-03-10 01:37:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:37:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:37:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:37:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:37:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-10 01:37:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 01:37:50 proc_lsst.multi:170 [INFO] [multi] job_id 16971853 proc_lsst.multi INFO: [multi] job_id 16971853 2024-03-10 01:37:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-10 01:37:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:50 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16971853 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16971853 parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 16971853 parsl.executors.status_handling INFO: Allocated block ID 13 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13 2024-03-10 01:37:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 2024-03-10 01:37:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:37:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:37:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:37:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:37:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-10 01:37:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 01:37:51 proc_lsst.multi:170 [INFO] [multi] job_id 16971854 proc_lsst.multi INFO: [multi] job_id 16971854 2024-03-10 01:37:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-10 01:37:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:51 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16971854 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16971854 parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 16971854 parsl.executors.status_handling INFO: Allocated block ID 14 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14 2024-03-10 01:37:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 2024-03-10 01:37:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:37:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:37:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:37:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:37:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-10 01:37:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 01:37:51 proc_lsst.multi:170 [INFO] [multi] job_id 16971855 proc_lsst.multi INFO: [multi] job_id 16971855 2024-03-10 01:37:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-10 01:37:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:51 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16971855 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16971855 parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 16971855 parsl.executors.status_handling INFO: Allocated block ID 15 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15 2024-03-10 01:37:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 2024-03-10 01:37:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:37:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:37:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:37:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:37:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-10 01:37:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 01:37:53 proc_lsst.multi:170 [INFO] [multi] job_id 16971856 proc_lsst.multi INFO: [multi] job_id 16971856 2024-03-10 01:37:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-10 01:37:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:53 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16971856 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16971856 parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 16971856 parsl.executors.status_handling INFO: Allocated block ID 16 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16 2024-03-10 01:37:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 2024-03-10 01:37:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:37:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:37:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:37:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:37:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-10 01:37:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 01:37:54 proc_lsst.multi:170 [INFO] [multi] job_id 16971857 proc_lsst.multi INFO: [multi] job_id 16971857 2024-03-10 01:37:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-10 01:37:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:54 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16971857 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16971857 parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 16971857 parsl.executors.status_handling INFO: Allocated block ID 17 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17 2024-03-10 01:37:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 2024-03-10 01:37:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:37:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:37:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:37:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:37:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-10 01:37:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 01:37:54 proc_lsst.multi:170 [INFO] [multi] job_id 16971858 proc_lsst.multi INFO: [multi] job_id 16971858 2024-03-10 01:37:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-10 01:37:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:54 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16971858 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16971858 parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 16971858 parsl.executors.status_handling INFO: Allocated block ID 18 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18 2024-03-10 01:37:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 2024-03-10 01:37:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:37:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:37:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:37:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:37:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-10 01:37:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 01:37:54 proc_lsst.multi:170 [INFO] [multi] job_id 16971859 proc_lsst.multi INFO: [multi] job_id 16971859 2024-03-10 01:37:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-10 01:37:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:54 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16971859 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16971859 parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 16971859 parsl.executors.status_handling INFO: Allocated block ID 19 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19 2024-03-10 01:37:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 2024-03-10 01:37:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:37:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:37:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:37:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:37:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-10 01:37:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 01:37:55 proc_lsst.multi:170 [INFO] [multi] job_id 16971860 proc_lsst.multi INFO: [multi] job_id 16971860 2024-03-10 01:37:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-10 01:37:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:55 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16971860 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16971860 parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 16971860 parsl.executors.status_handling INFO: Allocated block ID 20 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20 2024-03-10 01:37:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 2024-03-10 01:37:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:37:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:37:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:37:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:37:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-10 01:37:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 01:37:55 proc_lsst.multi:170 [INFO] [multi] job_id 16971861 proc_lsst.multi INFO: [multi] job_id 16971861 2024-03-10 01:37:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-10 01:37:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:55 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16971861 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16971861 parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 16971861 parsl.executors.status_handling INFO: Allocated block ID 21 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21 2024-03-10 01:37:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 2024-03-10 01:37:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:37:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:37:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:37:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:37:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-10 01:37:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 01:37:56 proc_lsst.multi:170 [INFO] [multi] job_id 16971862 proc_lsst.multi INFO: [multi] job_id 16971862 2024-03-10 01:37:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-10 01:37:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:56 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16971862 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16971862 parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 16971862 parsl.executors.status_handling INFO: Allocated block ID 22 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22 2024-03-10 01:37:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 2024-03-10 01:37:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:37:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:37:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:37:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:37:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-10 01:37:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 01:37:56 proc_lsst.multi:170 [INFO] [multi] job_id 16971863 proc_lsst.multi INFO: [multi] job_id 16971863 2024-03-10 01:37:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-10 01:37:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:56 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16971863 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16971863 parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 16971863 parsl.executors.status_handling INFO: Allocated block ID 23 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23 2024-03-10 01:37:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 2024-03-10 01:37:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:37:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:37:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:37:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:37:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-10 01:37:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 01:37:57 proc_lsst.multi:170 [INFO] [multi] job_id 16971864 proc_lsst.multi INFO: [multi] job_id 16971864 2024-03-10 01:37:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-10 01:37:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:57 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16971864 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16971864 parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 16971864 parsl.executors.status_handling INFO: Allocated block ID 24 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24 2024-03-10 01:37:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 2024-03-10 01:37:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:37:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:37:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:37:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:37:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-10 01:37:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 01:37:57 proc_lsst.multi:170 [INFO] [multi] job_id 16971865 proc_lsst.multi INFO: [multi] job_id 16971865 2024-03-10 01:37:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-10 01:37:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:57 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16971865 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16971865 parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 16971865 parsl.executors.status_handling INFO: Allocated block ID 25 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25 2024-03-10 01:37:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 2024-03-10 01:37:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:37:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:37:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:37:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:37:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-10 01:37:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 01:37:58 proc_lsst.multi:170 [INFO] [multi] job_id 16971866 proc_lsst.multi INFO: [multi] job_id 16971866 2024-03-10 01:37:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-10 01:37:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:58 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16971866 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16971866 parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 16971866 parsl.executors.status_handling INFO: Allocated block ID 26 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26 2024-03-10 01:37:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 2024-03-10 01:37:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:37:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:37:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:37:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:37:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-10 01:37:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 01:37:58 proc_lsst.multi:170 [INFO] [multi] job_id 16971867 proc_lsst.multi INFO: [multi] job_id 16971867 2024-03-10 01:37:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-10 01:37:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:58 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16971867 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16971867 parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 16971867 parsl.executors.status_handling INFO: Allocated block ID 27 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27 2024-03-10 01:37:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 2024-03-10 01:37:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:37:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:37:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:37:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:37:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-10 01:37:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 01:37:59 proc_lsst.multi:170 [INFO] [multi] job_id 16971868 proc_lsst.multi INFO: [multi] job_id 16971868 2024-03-10 01:37:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-10 01:37:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:59 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16971868 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16971868 parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 16971868 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23162860286032 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 28, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 1/27 running/pending blocks, and 5 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 28 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 19 more blocks parsl.executors.status_handling INFO: Scaling out by 19 blocks parsl.executors.status_handling INFO: Allocated block ID 28 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28 2024-03-10 01:37:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 2024-03-10 01:37:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:37:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:37:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:37:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:37:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-10 01:37:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 01:37:59 proc_lsst.multi:170 [INFO] [multi] job_id 16971869 proc_lsst.multi INFO: [multi] job_id 16971869 2024-03-10 01:37:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-10 01:37:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:59 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16971869 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16971869 parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 16971869 parsl.executors.status_handling INFO: Allocated block ID 29 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29 2024-03-10 01:37:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 2024-03-10 01:37:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:37:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:37:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:37:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:37:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-10 01:37:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:37:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 01:38:00 proc_lsst.multi:170 [INFO] [multi] job_id 16971870 proc_lsst.multi INFO: [multi] job_id 16971870 2024-03-10 01:38:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-10 01:38:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:38:00 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16971870 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16971870 parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 16971870 parsl.executors.status_handling INFO: Allocated block ID 30 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30 2024-03-10 01:38:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 2024-03-10 01:38:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:38:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:38:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:38:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:38:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-10 01:38:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:38:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5ba040057a58696c4253efd35e8797d with result from task 1 parsl.dataflow.dflow DEBUG: Task 1 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/92976e06-ac94-40e7-8e53-5e34106bd6a3_isrForCrosstalkSources_860801_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/92976e06-ac94-40e7-8e53-5e34106bd6a3_isrForCrosstalkSources_860801_49.stderr 2024-03-10 01:38:00 proc_lsst.multi:170 [INFO] [multi] job_id 16971871 proc_lsst.multi INFO: [multi] job_id 16971871 2024-03-10 01:38:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-10 01:38:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:38:00 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16971871 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16971871 parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 16971871 parsl.executors.status_handling INFO: Allocated block ID 31 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-31 2024-03-10 01:38:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 2024-03-10 01:38:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:38:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:38:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:38:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:38:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-10 01:38:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:38:00 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:38:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-10 01:38:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-10 01:38:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 01:38:01 proc_lsst.multi:170 [INFO] [multi] job_id 16971872 proc_lsst.multi INFO: [multi] job_id 16971872 2024-03-10 01:38:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:38:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-10 01:38:01 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16971872 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16971872 parsl.executors.status_handling DEBUG: Launched block 31 on executor multi with job ID 16971872 parsl.executors.status_handling INFO: Allocated block ID 32 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-32 2024-03-10 01:38:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 2024-03-10 01:38:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:38:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:38:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:38:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:38:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-10 01:38:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:38:01 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:38:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:38:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-10 01:38:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 01:38:01 proc_lsst.multi:170 [INFO] [multi] job_id 16971873 proc_lsst.multi INFO: [multi] job_id 16971873 2024-03-10 01:38:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-10 01:38:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-10 01:38:01 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16971873 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16971873 parsl.executors.status_handling DEBUG: Launched block 32 on executor multi with job ID 16971873 parsl.executors.status_handling INFO: Allocated block ID 33 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-33 2024-03-10 01:38:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 2024-03-10 01:38:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:38:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:38:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:38:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:38:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-10 01:38:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:38:01 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:38:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-10 01:38:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-10 01:38:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a5c6f0b1599d331ce367d16734b9b9a with result from task 0 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 0 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/84c2d1c4-9e10-43af-a259-94ad110d4c65_isrForCrosstalkSources_860803_44.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/84c2d1c4-9e10-43af-a259-94ad110d4c65_isrForCrosstalkSources_860803_44.stderr 2024-03-10 01:38:02 proc_lsst.multi:170 [INFO] [multi] job_id 16971874 proc_lsst.multi INFO: [multi] job_id 16971874 2024-03-10 01:38:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-10 01:38:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-10 01:38:02 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16971874 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16971874 parsl.executors.status_handling DEBUG: Launched block 33 on executor multi with job ID 16971874 parsl.executors.status_handling INFO: Allocated block ID 34 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-34 2024-03-10 01:38:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 2024-03-10 01:38:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:38:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:38:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:38:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:38:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-10 01:38:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:38:02 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:38:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-10 01:38:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-10 01:38:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 01:38:02 proc_lsst.multi:170 [INFO] [multi] job_id 16971875 proc_lsst.multi INFO: [multi] job_id 16971875 2024-03-10 01:38:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-10 01:38:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-10 01:38:02 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16971875 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16971875 parsl.executors.status_handling DEBUG: Launched block 34 on executor multi with job ID 16971875 parsl.executors.status_handling INFO: Allocated block ID 35 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-35 2024-03-10 01:38:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 2024-03-10 01:38:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:38:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:38:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:38:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:38:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-10 01:38:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:38:02 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:38:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-10 01:38:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-10 01:38:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 01:38:03 proc_lsst.multi:170 [INFO] [multi] job_id 16971877 proc_lsst.multi INFO: [multi] job_id 16971877 2024-03-10 01:38:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-10 01:38:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-10 01:38:03 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16971877 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16971877 parsl.executors.status_handling DEBUG: Launched block 35 on executor multi with job ID 16971877 parsl.executors.status_handling INFO: Allocated block ID 36 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-36 2024-03-10 01:38:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 2024-03-10 01:38:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:38:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:38:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:38:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:38:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-10 01:38:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:38:03 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:38:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-10 01:38:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-10 01:38:03 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:38:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-10 01:38:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-10 01:38:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 01:38:03 proc_lsst.multi:170 [INFO] [multi] job_id 16971878 proc_lsst.multi INFO: [multi] job_id 16971878 2024-03-10 01:38:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:38:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-10 01:38:03 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16971878 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16971878 parsl.executors.status_handling DEBUG: Launched block 36 on executor multi with job ID 16971878 parsl.executors.status_handling INFO: Allocated block ID 37 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-37 2024-03-10 01:38:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 2024-03-10 01:38:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:38:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:38:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:38:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:38:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-10 01:38:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:38:03 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:38:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-10 01:38:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-10 01:38:03 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:38:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:38:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-10 01:38:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 01:38:04 proc_lsst.multi:170 [INFO] [multi] job_id 16971879 proc_lsst.multi INFO: [multi] job_id 16971879 2024-03-10 01:38:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-10 01:38:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-10 01:38:04 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16971879 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16971879 parsl.executors.status_handling DEBUG: Launched block 37 on executor multi with job ID 16971879 parsl.executors.status_handling INFO: Allocated block ID 38 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-38 2024-03-10 01:38:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 2024-03-10 01:38:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:38:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:38:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:38:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:38:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-10 01:38:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:38:04 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:38:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-10 01:38:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-10 01:38:04 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:38:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-10 01:38:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-10 01:38:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30188158eb8e6e6e6980e6194255a1e1 with result from task 2 parsl.dataflow.dflow DEBUG: Task 2 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/11b36829-3f8c-4888-bfbe-254c3b6421bb_isrForCrosstalkSources_860803_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/11b36829-3f8c-4888-bfbe-254c3b6421bb_isrForCrosstalkSources_860803_8.stderr 2024-03-10 01:38:04 proc_lsst.multi:170 [INFO] [multi] job_id 16971880 proc_lsst.multi INFO: [multi] job_id 16971880 2024-03-10 01:38:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-10 01:38:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-10 01:38:04 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16971880 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16971880 parsl.executors.status_handling DEBUG: Launched block 38 on executor multi with job ID 16971880 parsl.executors.status_handling INFO: Allocated block ID 39 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-39 2024-03-10 01:38:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 2024-03-10 01:38:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:38:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:38:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:38:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:38:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-10 01:38:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:38:04 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:38:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-10 01:38:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-10 01:38:04 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:38:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-10 01:38:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-10 01:38:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 01:38:05 proc_lsst.multi:170 [INFO] [multi] job_id 16971881 proc_lsst.multi INFO: [multi] job_id 16971881 2024-03-10 01:38:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-10 01:38:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-10 01:38:05 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16971881 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16971881 parsl.executors.status_handling DEBUG: Launched block 39 on executor multi with job ID 16971881 parsl.executors.status_handling INFO: Allocated block ID 40 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-40 2024-03-10 01:38:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 2024-03-10 01:38:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:38:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:38:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:38:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:38:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-10 01:38:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:38:05 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:38:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-10 01:38:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-10 01:38:05 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:38:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-10 01:38:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-10 01:38:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 01:38:05 proc_lsst.multi:170 [INFO] [multi] job_id 16971882 proc_lsst.multi INFO: [multi] job_id 16971882 2024-03-10 01:38:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-10 01:38:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-10 01:38:05 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16971882 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16971882 parsl.executors.status_handling DEBUG: Launched block 40 on executor multi with job ID 16971882 parsl.executors.status_handling INFO: Allocated block ID 41 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-41 2024-03-10 01:38:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 2024-03-10 01:38:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:38:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:38:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:38:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:38:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-10 01:38:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:38:05 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:38:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-10 01:38:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-10 01:38:05 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:38:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-10 01:38:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-10 01:38:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 01:38:06 proc_lsst.multi:170 [INFO] [multi] job_id 16971883 proc_lsst.multi INFO: [multi] job_id 16971883 2024-03-10 01:38:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-10 01:38:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-10 01:38:06 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16971883 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16971883 parsl.executors.status_handling DEBUG: Launched block 41 on executor multi with job ID 16971883 parsl.executors.status_handling INFO: Allocated block ID 42 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-42 2024-03-10 01:38:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 2024-03-10 01:38:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:38:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:38:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:38:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:38:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-10 01:38:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:38:06 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:38:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-10 01:38:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-10 01:38:06 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:38:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-10 01:38:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-10 01:38:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 01:38:07 proc_lsst.multi:170 [INFO] [multi] job_id 16971884 proc_lsst.multi INFO: [multi] job_id 16971884 2024-03-10 01:38:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-10 01:38:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-10 01:38:07 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16971884 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16971884 parsl.executors.status_handling DEBUG: Launched block 42 on executor multi with job ID 16971884 parsl.executors.status_handling INFO: Allocated block ID 43 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-43 2024-03-10 01:38:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 2024-03-10 01:38:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:38:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:38:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:38:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:38:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-10 01:38:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:38:07 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:38:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-10 01:38:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-10 01:38:07 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:38:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-10 01:38:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-10 01:38:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 01:38:07 proc_lsst.multi:170 [INFO] [multi] job_id 16971885 proc_lsst.multi INFO: [multi] job_id 16971885 2024-03-10 01:38:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-10 01:38:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-10 01:38:07 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16971885 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16971885 parsl.executors.status_handling DEBUG: Launched block 43 on executor multi with job ID 16971885 parsl.executors.status_handling INFO: Allocated block ID 44 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-44 2024-03-10 01:38:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 2024-03-10 01:38:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:38:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:38:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:38:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:38:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-10 01:38:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:38:07 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:38:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-10 01:38:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-10 01:38:07 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:38:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-10 01:38:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-10 01:38:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 01:38:08 proc_lsst.multi:170 [INFO] [multi] job_id 16971886 proc_lsst.multi INFO: [multi] job_id 16971886 2024-03-10 01:38:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-10 01:38:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-10 01:38:08 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16971886 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16971886 parsl.executors.status_handling DEBUG: Launched block 44 on executor multi with job ID 16971886 parsl.executors.status_handling INFO: Allocated block ID 45 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-45 2024-03-10 01:38:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 2024-03-10 01:38:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:38:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:38:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:38:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:38:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-10 01:38:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:38:08 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:38:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-10 01:38:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-10 01:38:08 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:38:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-10 01:38:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-10 01:38:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 01:38:08 proc_lsst.multi:170 [INFO] [multi] job_id 16971887 proc_lsst.multi INFO: [multi] job_id 16971887 2024-03-10 01:38:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-10 01:38:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-10 01:38:08 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16971887 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16971887 parsl.executors.status_handling DEBUG: Launched block 45 on executor multi with job ID 16971887 parsl.executors.status_handling INFO: Allocated block ID 46 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-46 2024-03-10 01:38:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 2024-03-10 01:38:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-10 01:38:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-10 01:38:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-10 01:38:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:38:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-10 01:38:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-10 01:38:08 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:38:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-10 01:38:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-10 01:38:08 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-10 01:38:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-10 01:38:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-10 01:38:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3010,127.0.0.1,10.64.65.10,169.254.95.120,198.48.92.26,10.64.129.10 -p 0 -c 1.0 -m None --poll 10 --task_port=54287 --result_port=54397 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-10 01:38:09 proc_lsst.multi:170 [INFO] [multi] job_id 16971888 proc_lsst.multi INFO: [multi] job_id 16971888 2024-03-10 01:38:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-10 01:38:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-10 01:38:09 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16971888 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16971888 parsl.executors.status_handling DEBUG: Launched block 46 on executor multi with job ID 16971888 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23162860286032 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 183 parsl.dataflow.strategy DEBUG: Executor multi has 183 active tasks, 1/46 running/pending blocks, and 8 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23162860286032 parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3666efbbf11bbba35565bd0f2507f54 with result from task 3 parsl.dataflow.dflow DEBUG: Task 3 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/8d557ce3-135a-4574-ae31-b3b20698843f_isrForCrosstalkSources_860803_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/8d557ce3-135a-4574-ae31-b3b20698843f_isrForCrosstalkSources_860803_42.stderr parsl.dataflow.dflow INFO: Task 5 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f478d9dce681723d2e4fe44147c1c1d0 with result from task 5 parsl.dataflow.dflow DEBUG: Task 5 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/1cabc528-c495-4004-ae61-f26f33ccd1dd_isrForCrosstalkSources_860803_47.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/1cabc528-c495-4004-ae61-f26f33ccd1dd_isrForCrosstalkSources_860803_47.stderr parsl.dataflow.dflow INFO: Task 4 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5718219c26222edd42d4d7b93a79d6fa with result from task 4 parsl.dataflow.dflow DEBUG: Task 4 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/5b12376e-5c99-459c-9034-e707294ef90d_isrForCrosstalkSources_860802_28.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/5b12376e-5c99-459c-9034-e707294ef90d_isrForCrosstalkSources_860802_28.stderr parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e40d4dceccdbaf27a477b17de7548969 with result from task 6 parsl.dataflow.dflow DEBUG: Task 6 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/69ab5d8b-3c5d-4296-a84f-d9178bd6c13c_isrForCrosstalkSources_860802_42.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/69ab5d8b-3c5d-4296-a84f-d9178bd6c13c_isrForCrosstalkSources_860802_42.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 179 parsl.dataflow.strategy DEBUG: Executor multi has 179 active tasks, 1/46 running/pending blocks, and 8 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23162860286032 parsl.dataflow.dflow INFO: Task 7 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dba9d4c981b8db89270f7dd1fd8449e0 with result from task 7 parsl.dataflow.dflow DEBUG: Task 7 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/8017b810-b22f-462c-82e1-52addb06ba8c_isrForCrosstalkSources_860803_5.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/8017b810-b22f-462c-82e1-52addb06ba8c_isrForCrosstalkSources_860803_5.stderr 2024-03-10 01:38:19 proc_lsst.multi:146 [INFO] found job 44401 in provider local proc_lsst.multi INFO: found job 44401 in provider local 2024-03-10 01:38:19 proc_lsst.multi:146 [INFO] found job 16971842 in provider astro proc_lsst.multi INFO: found job 16971842 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971871 PD 16971870 PD 16971869 PD 16971868 PD 16971867 PD 16971866 PD 16971865 PD 16971864 PD 16971863 PD 16971862 PD 16971861 PD 16971860 PD 16971859 PD 16971858 PD 16971857 PD 16971856 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:19 proc_lsst.multi:146 [INFO] found job 16971843 in provider astro proc_lsst.multi INFO: found job 16971843 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971871 PD 16971870 PD 16971869 PD 16971868 PD 16971867 PD 16971866 PD 16971865 PD 16971864 PD 16971863 PD 16971862 PD 16971861 PD 16971860 PD 16971859 PD 16971858 PD 16971857 PD 16971856 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:20 proc_lsst.multi:146 [INFO] found job 16971844 in provider astro proc_lsst.multi INFO: found job 16971844 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971871 PD 16971870 PD 16971869 PD 16971868 PD 16971867 PD 16971866 PD 16971865 PD 16971864 PD 16971863 PD 16971862 PD 16971861 PD 16971860 PD 16971859 PD 16971858 PD 16971857 PD 16971856 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:20 proc_lsst.multi:146 [INFO] found job 16971845 in provider astro proc_lsst.multi INFO: found job 16971845 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971871 PD 16971870 PD 16971869 PD 16971868 PD 16971867 PD 16971866 PD 16971865 PD 16971864 PD 16971863 PD 16971862 PD 16971861 PD 16971860 PD 16971859 PD 16971858 PD 16971857 PD 16971856 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:21 proc_lsst.multi:146 [INFO] found job 16971846 in provider astro proc_lsst.multi INFO: found job 16971846 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971871 PD 16971870 PD 16971869 PD 16971868 PD 16971867 PD 16971866 PD 16971865 PD 16971864 PD 16971863 PD 16971862 PD 16971861 PD 16971860 PD 16971859 PD 16971858 PD 16971857 PD 16971856 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:21 proc_lsst.multi:146 [INFO] found job 16971847 in provider astro proc_lsst.multi INFO: found job 16971847 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971871 PD 16971870 PD 16971869 PD 16971868 PD 16971867 PD 16971866 PD 16971865 PD 16971864 PD 16971863 PD 16971862 PD 16971861 PD 16971860 PD 16971859 PD 16971858 PD 16971857 PD 16971856 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:22 proc_lsst.multi:146 [INFO] found job 16971848 in provider astro proc_lsst.multi INFO: found job 16971848 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971871 PD 16971870 PD 16971869 PD 16971868 PD 16971867 PD 16971866 PD 16971865 PD 16971864 PD 16971863 PD 16971862 PD 16971861 PD 16971860 PD 16971859 PD 16971858 PD 16971857 PD 16971856 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:23 proc_lsst.multi:146 [INFO] found job 16971849 in provider astro proc_lsst.multi INFO: found job 16971849 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971871 PD 16971870 PD 16971869 PD 16971868 PD 16971867 PD 16971866 PD 16971865 PD 16971864 PD 16971863 PD 16971862 PD 16971861 PD 16971860 PD 16971859 PD 16971858 PD 16971857 PD 16971856 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:23 proc_lsst.multi:146 [INFO] found job 16971850 in provider astro proc_lsst.multi INFO: found job 16971850 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971871 PD 16971870 PD 16971869 PD 16971868 PD 16971867 PD 16971866 PD 16971865 PD 16971864 PD 16971863 PD 16971862 PD 16971861 PD 16971860 PD 16971859 PD 16971858 PD 16971857 PD 16971856 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:24 proc_lsst.multi:146 [INFO] found job 16971851 in provider astro proc_lsst.multi INFO: found job 16971851 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b5d4022490d69ffa149d5aa1a09cfba with result from task 12 parsl.dataflow.dflow DEBUG: Task 12 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/05bb4e71-5687-466f-8361-2caefd851eab_isrForCrosstalkSources_860803_34.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/05bb4e71-5687-466f-8361-2caefd851eab_isrForCrosstalkSources_860803_34.stderr parsl.dataflow.dflow INFO: Task 11 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a5af4add3cd845dedb14f69d1c42f08 with result from task 11 parsl.dataflow.dflow DEBUG: Task 11 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/fe4e406a-f54a-466e-b94c-cc3babe3ea73_isrForCrosstalkSources_860803_22.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/fe4e406a-f54a-466e-b94c-cc3babe3ea73_isrForCrosstalkSources_860803_22.stderr parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 210752207651ea9904ad9c734e8d9952 with result from task 14 parsl.dataflow.dflow DEBUG: Task 14 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/f61c3c66-643a-4e18-a16c-1cf82e6afd29_isrForCrosstalkSources_860801_41.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/f61c3c66-643a-4e18-a16c-1cf82e6afd29_isrForCrosstalkSources_860801_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:26 proc_lsst.multi:146 [INFO] found job 16971852 in provider astro proc_lsst.multi INFO: found job 16971852 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 427be0ce20ee88a02a56f281c51c669b with result from task 15 parsl.dataflow.dflow DEBUG: Task 15 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/f0c55dbf-7437-49b6-bdeb-f5ddcbf19439_isrForCrosstalkSources_860803_4.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/f0c55dbf-7437-49b6-bdeb-f5ddcbf19439_isrForCrosstalkSources_860803_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:26 proc_lsst.multi:146 [INFO] found job 16971853 in provider astro proc_lsst.multi INFO: found job 16971853 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:27 proc_lsst.multi:146 [INFO] found job 16971854 in provider astro proc_lsst.multi INFO: found job 16971854 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:27 proc_lsst.multi:146 [INFO] found job 16971855 in provider astro proc_lsst.multi INFO: found job 16971855 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:28 proc_lsst.multi:146 [INFO] found job 16971856 in provider astro proc_lsst.multi INFO: found job 16971856 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:28 proc_lsst.multi:146 [INFO] found job 16971857 in provider astro proc_lsst.multi INFO: found job 16971857 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b320d1b8465d42fe1b35e6e0550fee8 with result from task 13 parsl.dataflow.dflow DEBUG: Task 13 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/7855529c-dc41-4801-ac53-c497ecc159ff_isrForCrosstalkSources_860803_9.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/7855529c-dc41-4801-ac53-c497ecc159ff_isrForCrosstalkSources_860803_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:29 proc_lsst.multi:146 [INFO] found job 16971858 in provider astro proc_lsst.multi INFO: found job 16971858 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:30 proc_lsst.multi:146 [INFO] found job 16971859 in provider astro proc_lsst.multi INFO: found job 16971859 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:30 proc_lsst.multi:146 [INFO] found job 16971860 in provider astro proc_lsst.multi INFO: found job 16971860 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:31 proc_lsst.multi:146 [INFO] found job 16971861 in provider astro proc_lsst.multi INFO: found job 16971861 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:31 proc_lsst.multi:146 [INFO] found job 16971862 in provider astro proc_lsst.multi INFO: found job 16971862 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:32 proc_lsst.multi:146 [INFO] found job 16971863 in provider astro proc_lsst.multi INFO: found job 16971863 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:32 proc_lsst.multi:146 [INFO] found job 16971864 in provider astro proc_lsst.multi INFO: found job 16971864 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:33 proc_lsst.multi:146 [INFO] found job 16971865 in provider astro proc_lsst.multi INFO: found job 16971865 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:33 proc_lsst.multi:146 [INFO] found job 16971866 in provider astro proc_lsst.multi INFO: found job 16971866 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:34 proc_lsst.multi:146 [INFO] found job 16971867 in provider astro proc_lsst.multi INFO: found job 16971867 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:34 proc_lsst.multi:146 [INFO] found job 16971868 in provider astro proc_lsst.multi INFO: found job 16971868 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:35 proc_lsst.multi:146 [INFO] found job 16971869 in provider astro proc_lsst.multi INFO: found job 16971869 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:35 proc_lsst.multi:146 [INFO] found job 16971870 in provider astro proc_lsst.multi INFO: found job 16971870 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:36 proc_lsst.multi:146 [INFO] found job 16971871 in provider astro proc_lsst.multi INFO: found job 16971871 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:36 proc_lsst.multi:146 [INFO] found job 16971872 in provider escience_rtx6k proc_lsst.multi INFO: found job 16971872 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971872,16971873,16971874,16971875,16971877' parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0fad56391e23cfa49d656b9c145b17b9 with result from task 24 parsl.dataflow.dflow DEBUG: Task 24 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/ced5c32d-8c38-4cb9-8640-85765b3fd23f_isrForCrosstalkSources_860803_12.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/ced5c32d-8c38-4cb9-8640-85765b3fd23f_isrForCrosstalkSources_860803_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971877 R 16971872 R 16971873 R 16971874 R 16971875 R parsl.providers.slurm.slurm DEBUG: Updating job 16971877 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971872 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971873 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971874 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971875 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:37 proc_lsst.multi:146 [INFO] found job 16971873 in provider escience_rtx6k proc_lsst.multi INFO: found job 16971873 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971872,16971873,16971874,16971875,16971877' parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d45d5ad6882089137554410f081dab2a with result from task 27 parsl.dataflow.dflow DEBUG: Task 27 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/a7334b40-6503-485d-9143-123435c02d2d_isrForCrosstalkSources_860801_43.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/a7334b40-6503-485d-9143-123435c02d2d_isrForCrosstalkSources_860801_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971877 R 16971872 R 16971873 R 16971874 R 16971875 R parsl.providers.slurm.slurm DEBUG: Updating job 16971877 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971872 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971873 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971874 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971875 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:38 proc_lsst.multi:146 [INFO] found job 16971874 in provider escience_rtx6k proc_lsst.multi INFO: found job 16971874 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971872,16971873,16971874,16971875,16971877' parsl.providers.slurm.slurm DEBUG: squeue returned 16971877 R 16971872 R 16971873 R 16971874 R 16971875 R parsl.providers.slurm.slurm DEBUG: Updating job 16971877 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971872 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971873 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971874 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971875 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:38 proc_lsst.multi:146 [INFO] found job 16971875 in provider escience_rtx6k proc_lsst.multi INFO: found job 16971875 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971872,16971873,16971874,16971875,16971877' parsl.providers.slurm.slurm DEBUG: squeue returned 16971877 R 16971872 R 16971873 R 16971874 R 16971875 R parsl.providers.slurm.slurm DEBUG: Updating job 16971877 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971872 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971873 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971874 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971875 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:38 proc_lsst.multi:146 [INFO] found job 16971877 in provider escience_rtx6k proc_lsst.multi INFO: found job 16971877 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971872,16971873,16971874,16971875,16971877' parsl.providers.slurm.slurm DEBUG: squeue returned 16971877 R 16971872 R 16971873 R 16971874 R 16971875 R parsl.providers.slurm.slurm DEBUG: Updating job 16971877 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971872 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971873 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971874 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971875 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:39 proc_lsst.multi:146 [INFO] found job 16971878 in provider escience_a40 proc_lsst.multi INFO: found job 16971878 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.dataflow.dflow INFO: Task 26 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 432d9f0abd9a51a3fbe30d1c692f18d3 with result from task 26 parsl.dataflow.dflow DEBUG: Task 26 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/c41fa473-f795-4cab-bf40-8ba0a6618ab1_isrForCrosstalkSources_860803_14.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/c41fa473-f795-4cab-bf40-8ba0a6618ab1_isrForCrosstalkSources_860803_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:40 proc_lsst.multi:146 [INFO] found job 16971879 in provider escience_a40 proc_lsst.multi INFO: found job 16971879 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:40 proc_lsst.multi:146 [INFO] found job 16971880 in provider escience_a40 proc_lsst.multi INFO: found job 16971880 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.dataflow.dflow INFO: Task 25 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b85694b0fcf7b5f58a8f995475e5b812 with result from task 25 parsl.dataflow.dflow DEBUG: Task 25 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/72b58ae1-bc9b-4472-97e3-15487ab9428f_isrForCrosstalkSources_860802_47.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/72b58ae1-bc9b-4472-97e3-15487ab9428f_isrForCrosstalkSources_860802_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:41 proc_lsst.multi:146 [INFO] found job 16971881 in provider escience_a40 proc_lsst.multi INFO: found job 16971881 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:41 proc_lsst.multi:146 [INFO] found job 16971882 in provider escience_a40 proc_lsst.multi INFO: found job 16971882 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:42 proc_lsst.multi:146 [INFO] found job 16971883 in provider escience_a40 proc_lsst.multi INFO: found job 16971883 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:42 proc_lsst.multi:146 [INFO] found job 16971884 in provider escience_a40 proc_lsst.multi INFO: found job 16971884 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:43 proc_lsst.multi:146 [INFO] found job 16971885 in provider escience_a40 proc_lsst.multi INFO: found job 16971885 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.dataflow.dflow INFO: Task 28 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4a2325c17666388e2d2e639ca2b70ca with result from task 28 parsl.dataflow.dflow DEBUG: Task 28 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/7e989704-dc7f-4d0f-9432-ea1c30d727af_isrForCrosstalkSources_860801_11.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/7e989704-dc7f-4d0f-9432-ea1c30d727af_isrForCrosstalkSources_860801_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:43 proc_lsst.multi:146 [INFO] found job 16971886 in provider escience_a40 proc_lsst.multi INFO: found job 16971886 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:44 proc_lsst.multi:146 [INFO] found job 16971887 in provider escience_a40 proc_lsst.multi INFO: found job 16971887 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:44 proc_lsst.multi:146 [INFO] found job 16971888 in provider escience_a40 proc_lsst.multi INFO: found job 16971888 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 168 parsl.dataflow.strategy DEBUG: Executor multi has 168 active tasks, 23/24 running/pending blocks, and 23 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23162860286032 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 168 parsl.dataflow.strategy DEBUG: Executor multi has 168 active tasks, 23/24 running/pending blocks, and 23 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23162860286032 parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 575a9f20e622e82ba21fc988298a77c3 with result from task 30 parsl.dataflow.dflow DEBUG: Task 30 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/d2716afc-c266-4939-aa6c-b7d1ddd06c53_isrForCrosstalkSources_860803_15.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/d2716afc-c266-4939-aa6c-b7d1ddd06c53_isrForCrosstalkSources_860803_15.stderr parsl.dataflow.dflow INFO: Task 29 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94cc5b62a9f97de53824d89752c57344 with result from task 29 parsl.dataflow.dflow DEBUG: Task 29 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/6396ea2b-ea74-45d5-bc1c-df8445a13d86_isrForCrosstalkSources_860803_54.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/6396ea2b-ea74-45d5-bc1c-df8445a13d86_isrForCrosstalkSources_860803_54.stderr 2024-03-10 01:38:50 proc_lsst.multi:146 [INFO] found job 44401 in provider local proc_lsst.multi INFO: found job 44401 in provider local 2024-03-10 01:38:50 proc_lsst.multi:146 [INFO] found job 16971842 in provider astro proc_lsst.multi INFO: found job 16971842 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:50 proc_lsst.multi:146 [INFO] found job 16971843 in provider astro proc_lsst.multi INFO: found job 16971843 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:50 proc_lsst.multi:146 [INFO] found job 16971844 in provider astro proc_lsst.multi INFO: found job 16971844 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:50 proc_lsst.multi:146 [INFO] found job 16971845 in provider astro proc_lsst.multi INFO: found job 16971845 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:51 proc_lsst.multi:146 [INFO] found job 16971846 in provider astro proc_lsst.multi INFO: found job 16971846 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:51 proc_lsst.multi:146 [INFO] found job 16971847 in provider astro proc_lsst.multi INFO: found job 16971847 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:51 proc_lsst.multi:146 [INFO] found job 16971848 in provider astro proc_lsst.multi INFO: found job 16971848 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:52 proc_lsst.multi:146 [INFO] found job 16971849 in provider astro proc_lsst.multi INFO: found job 16971849 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.dataflow.dflow INFO: Task 31 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93bfe4cf6db9e8e8b3a6c2f9a55a714d with result from task 31 parsl.dataflow.dflow DEBUG: Task 31 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/9eddd923-8e43-4f35-8ddc-32b8fe00e085_isrForCrosstalkSources_860801_45.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/9eddd923-8e43-4f35-8ddc-32b8fe00e085_isrForCrosstalkSources_860801_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:52 proc_lsst.multi:146 [INFO] found job 16971850 in provider astro proc_lsst.multi INFO: found job 16971850 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:53 proc_lsst.multi:146 [INFO] found job 16971851 in provider astro proc_lsst.multi INFO: found job 16971851 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:53 proc_lsst.multi:146 [INFO] found job 16971852 in provider astro proc_lsst.multi INFO: found job 16971852 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:53 proc_lsst.multi:146 [INFO] found job 16971853 in provider astro proc_lsst.multi INFO: found job 16971853 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:53 proc_lsst.multi:146 [INFO] found job 16971854 in provider astro proc_lsst.multi INFO: found job 16971854 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:54 proc_lsst.multi:146 [INFO] found job 16971855 in provider astro proc_lsst.multi INFO: found job 16971855 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:54 proc_lsst.multi:146 [INFO] found job 16971856 in provider astro proc_lsst.multi INFO: found job 16971856 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:54 proc_lsst.multi:146 [INFO] found job 16971857 in provider astro proc_lsst.multi INFO: found job 16971857 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:55 proc_lsst.multi:146 [INFO] found job 16971858 in provider astro proc_lsst.multi INFO: found job 16971858 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:55 proc_lsst.multi:146 [INFO] found job 16971859 in provider astro proc_lsst.multi INFO: found job 16971859 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:56 proc_lsst.multi:146 [INFO] found job 16971860 in provider astro proc_lsst.multi INFO: found job 16971860 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.dataflow.dflow INFO: Task 32 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df01b2d10b1a1404b86678b006c53d67 with result from task 32 parsl.dataflow.dflow DEBUG: Task 32 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/652b21af-a4cd-427d-b30e-85e5156a7a54_isrForCrosstalkSources_860801_27.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/652b21af-a4cd-427d-b30e-85e5156a7a54_isrForCrosstalkSources_860801_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:58 proc_lsst.multi:146 [INFO] found job 16971861 in provider astro proc_lsst.multi INFO: found job 16971861 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:59 proc_lsst.multi:146 [INFO] found job 16971862 in provider astro proc_lsst.multi INFO: found job 16971862 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:38:59 proc_lsst.multi:146 [INFO] found job 16971863 in provider astro proc_lsst.multi INFO: found job 16971863 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:00 proc_lsst.multi:146 [INFO] found job 16971864 in provider astro proc_lsst.multi INFO: found job 16971864 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:00 proc_lsst.multi:146 [INFO] found job 16971865 in provider astro proc_lsst.multi INFO: found job 16971865 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:00 proc_lsst.multi:146 [INFO] found job 16971866 in provider astro proc_lsst.multi INFO: found job 16971866 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02b7dc4f0d39d1b20c1f40a5778865b9 with result from task 33 parsl.dataflow.dflow DEBUG: Task 33 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/27e9085d-e0b4-4154-9262-b353b57807ac_isrForCrosstalkSources_860801_1.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/27e9085d-e0b4-4154-9262-b353b57807ac_isrForCrosstalkSources_860801_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:01 proc_lsst.multi:146 [INFO] found job 16971867 in provider astro proc_lsst.multi INFO: found job 16971867 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:01 proc_lsst.multi:146 [INFO] found job 16971868 in provider astro proc_lsst.multi INFO: found job 16971868 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.dataflow.dflow INFO: Task 41 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab415f3cf90801d0673e527736bfe5c3 with result from task 41 parsl.dataflow.dflow DEBUG: Task 41 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/b494bdd2-0a27-4e7e-aecc-90c23b64be55_isrForCrosstalkSources_860802_21.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/b494bdd2-0a27-4e7e-aecc-90c23b64be55_isrForCrosstalkSources_860802_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:02 proc_lsst.multi:146 [INFO] found job 16971869 in provider astro proc_lsst.multi INFO: found job 16971869 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:02 proc_lsst.multi:146 [INFO] found job 16971870 in provider astro proc_lsst.multi INFO: found job 16971870 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:02 proc_lsst.multi:146 [INFO] found job 16971871 in provider astro proc_lsst.multi INFO: found job 16971871 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971848 PD 16971849 PD 16971850 PD 16971851 PD 16971852 PD 16971853 PD 16971854 PD 16971855 PD 16971856 PD 16971857 PD 16971858 PD 16971859 PD 16971860 PD 16971861 PD 16971862 PD 16971863 PD 16971864 PD 16971865 PD 16971866 PD 16971867 PD 16971868 PD 16971869 PD 16971870 PD 16971871 PD 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:03 proc_lsst.multi:146 [INFO] found job 16971872 in provider escience_rtx6k proc_lsst.multi INFO: found job 16971872 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971872,16971873,16971874,16971875,16971877' parsl.dataflow.dflow INFO: Task 42 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f1991707abedc75f0f63389726438b7a with result from task 42 parsl.dataflow.dflow DEBUG: Task 42 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/44bc8a8e-b038-435d-a2e0-df5ed2749f7b_isrForCrosstalkSources_860801_31.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/44bc8a8e-b038-435d-a2e0-df5ed2749f7b_isrForCrosstalkSources_860801_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971877 R 16971872 R 16971873 R 16971874 R 16971875 R parsl.providers.slurm.slurm DEBUG: Updating job 16971877 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971872 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971873 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971874 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971875 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:03 proc_lsst.multi:146 [INFO] found job 16971873 in provider escience_rtx6k proc_lsst.multi INFO: found job 16971873 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971872,16971873,16971874,16971875,16971877' parsl.providers.slurm.slurm DEBUG: squeue returned 16971877 R 16971872 R 16971873 R 16971874 R 16971875 R parsl.providers.slurm.slurm DEBUG: Updating job 16971877 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971872 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971873 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971874 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971875 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:04 proc_lsst.multi:146 [INFO] found job 16971874 in provider escience_rtx6k proc_lsst.multi INFO: found job 16971874 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971872,16971873,16971874,16971875,16971877' parsl.providers.slurm.slurm DEBUG: squeue returned 16971877 R 16971872 R 16971873 R 16971874 R 16971875 R parsl.providers.slurm.slurm DEBUG: Updating job 16971877 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971872 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971873 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971874 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971875 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:04 proc_lsst.multi:146 [INFO] found job 16971875 in provider escience_rtx6k proc_lsst.multi INFO: found job 16971875 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971872,16971873,16971874,16971875,16971877' parsl.providers.slurm.slurm DEBUG: squeue returned 16971877 R 16971872 R 16971873 R 16971874 R 16971875 R parsl.providers.slurm.slurm DEBUG: Updating job 16971877 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971872 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971873 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971874 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971875 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:04 proc_lsst.multi:146 [INFO] found job 16971877 in provider escience_rtx6k proc_lsst.multi INFO: found job 16971877 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971872,16971873,16971874,16971875,16971877' parsl.providers.slurm.slurm DEBUG: squeue returned 16971877 R 16971872 R 16971873 R 16971874 R 16971875 R parsl.providers.slurm.slurm DEBUG: Updating job 16971877 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971872 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971873 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971874 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971875 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:04 proc_lsst.multi:146 [INFO] found job 16971878 in provider escience_a40 proc_lsst.multi INFO: found job 16971878 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:05 proc_lsst.multi:146 [INFO] found job 16971879 in provider escience_a40 proc_lsst.multi INFO: found job 16971879 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:05 proc_lsst.multi:146 [INFO] found job 16971880 in provider escience_a40 proc_lsst.multi INFO: found job 16971880 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:06 proc_lsst.multi:146 [INFO] found job 16971881 in provider escience_a40 proc_lsst.multi INFO: found job 16971881 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:06 proc_lsst.multi:146 [INFO] found job 16971882 in provider escience_a40 proc_lsst.multi INFO: found job 16971882 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:06 proc_lsst.multi:146 [INFO] found job 16971883 in provider escience_a40 proc_lsst.multi INFO: found job 16971883 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:07 proc_lsst.multi:146 [INFO] found job 16971884 in provider escience_a40 proc_lsst.multi INFO: found job 16971884 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:07 proc_lsst.multi:146 [INFO] found job 16971885 in provider escience_a40 proc_lsst.multi INFO: found job 16971885 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:07 proc_lsst.multi:146 [INFO] found job 16971886 in provider escience_a40 proc_lsst.multi INFO: found job 16971886 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:08 proc_lsst.multi:146 [INFO] found job 16971887 in provider escience_a40 proc_lsst.multi INFO: found job 16971887 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:08 proc_lsst.multi:146 [INFO] found job 16971888 in provider escience_a40 proc_lsst.multi INFO: found job 16971888 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.dataflow.dflow INFO: Task 43 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b39de7e8e7970e4a644c11343c310996 with result from task 43 parsl.dataflow.dflow DEBUG: Task 43 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/10d0da8c-fb6c-418e-b160-446775441c01_isrForCrosstalkSources_860801_19.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/10d0da8c-fb6c-418e-b160-446775441c01_isrForCrosstalkSources_860801_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 160 parsl.dataflow.strategy DEBUG: Executor multi has 160 active tasks, 23/24 running/pending blocks, and 24 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23162860286032 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 160 parsl.dataflow.strategy DEBUG: Executor multi has 160 active tasks, 23/24 running/pending blocks, and 24 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23162860286032 parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79bdd93a1ccb72697a1e02d348048f1c with result from task 9 parsl.dataflow.dflow DEBUG: Task 9 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/0ef0e551-bc62-47fa-a6f4-207e31e25040_isrForCrosstalkSources_860802_55.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/0ef0e551-bc62-47fa-a6f4-207e31e25040_isrForCrosstalkSources_860802_55.stderr parsl.dataflow.dflow INFO: Task 10 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f01a52fb556210f11540184fb14af3be with result from task 10 parsl.dataflow.dflow DEBUG: Task 10 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/27f97124-4191-4a9e-9332-a8b519369327_isrForCrosstalkSources_860802_60.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/27f97124-4191-4a9e-9332-a8b519369327_isrForCrosstalkSources_860802_60.stderr parsl.dataflow.dflow INFO: Task 8 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95627e28428e55d36da7d66f22d9aeb5 with result from task 8 parsl.dataflow.dflow DEBUG: Task 8 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/2e20cfae-845e-4f5e-ae0b-8364d1ab7914_isrForCrosstalkSources_860802_35.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/2e20cfae-845e-4f5e-ae0b-8364d1ab7914_isrForCrosstalkSources_860802_35.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 157 parsl.dataflow.strategy DEBUG: Executor multi has 157 active tasks, 23/24 running/pending blocks, and 24 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23162860286032 parsl.dataflow.dflow INFO: Task 47 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2932e634f975f0fc873d5f724cf35b9 with result from task 47 parsl.dataflow.dflow DEBUG: Task 47 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/a1ccf741-a2ba-4004-af1f-a16c669b6a37_isrForCrosstalkSources_860801_17.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/a1ccf741-a2ba-4004-af1f-a16c669b6a37_isrForCrosstalkSources_860801_17.stderr parsl.dataflow.dflow INFO: Task 48 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98828376025df308d0c5556eecc3099f with result from task 48 parsl.dataflow.dflow DEBUG: Task 48 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/afb1d44f-f0f6-46df-9e5c-5bb2094e095d_isrForCrosstalkSources_860802_33.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/afb1d44f-f0f6-46df-9e5c-5bb2094e095d_isrForCrosstalkSources_860802_33.stderr parsl.dataflow.dflow INFO: Task 46 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f3c8aecc01fc0e27234ec682e8d1a07 with result from task 46 parsl.dataflow.dflow DEBUG: Task 46 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/90d7d673-3f92-41c4-95a4-25d586f0483b_isrForCrosstalkSources_860802_12.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/90d7d673-3f92-41c4-95a4-25d586f0483b_isrForCrosstalkSources_860802_12.stderr parsl.dataflow.dflow INFO: Task 44 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea78c28da6ad031223df889bc9cff126 with result from task 44 parsl.dataflow.dflow DEBUG: Task 44 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/c1368ff9-cb09-46c9-87a9-db1cb28d6801_isrForCrosstalkSources_860802_37.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/c1368ff9-cb09-46c9-87a9-db1cb28d6801_isrForCrosstalkSources_860802_37.stderr parsl.dataflow.dflow INFO: Task 19 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d396bfe43cfbcef2dc70a48f472e7766 with result from task 19 parsl.dataflow.dflow DEBUG: Task 19 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/7c26a2fa-f35f-4a3f-9286-dbdf28f33e1b_isrForCrosstalkSources_860801_23.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/7c26a2fa-f35f-4a3f-9286-dbdf28f33e1b_isrForCrosstalkSources_860801_23.stderr parsl.dataflow.dflow INFO: Task 20 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf630881a8235ba197368a3386fb2202 with result from task 20 parsl.dataflow.dflow DEBUG: Task 20 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/18b9e2b8-cf10-4ef6-858e-74317578427c_isrForCrosstalkSources_860801_7.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/18b9e2b8-cf10-4ef6-858e-74317578427c_isrForCrosstalkSources_860801_7.stderr parsl.dataflow.dflow INFO: Task 23 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0cf734a078bf12c8b36c84c15ae9290 with result from task 23 parsl.dataflow.dflow DEBUG: Task 23 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/5b1052a3-2b2b-48c9-a923-003bd40699d7_isrForCrosstalkSources_860802_1.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/5b1052a3-2b2b-48c9-a923-003bd40699d7_isrForCrosstalkSources_860802_1.stderr parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca7b7bf53a084d55391e9fb834098610 with result from task 21 parsl.dataflow.dflow DEBUG: Task 21 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/5a0d6b15-e893-4565-af8d-8f9fab96aaf4_isrForCrosstalkSources_860802_62.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/5a0d6b15-e893-4565-af8d-8f9fab96aaf4_isrForCrosstalkSources_860802_62.stderr parsl.dataflow.dflow INFO: Task 22 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1e2fa1b5e318d7a746996f48bf48f08 with result from task 22 parsl.dataflow.dflow DEBUG: Task 22 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/5c609f61-cae9-4b03-8b32-2b76452a6404_isrForCrosstalkSources_860801_62.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/5c609f61-cae9-4b03-8b32-2b76452a6404_isrForCrosstalkSources_860801_62.stderr parsl.dataflow.dflow INFO: Task 17 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95143b39cccd2b5dfa77332aec33dfd9 with result from task 17 parsl.dataflow.dflow DEBUG: Task 17 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/69388045-56f7-4cb7-8ecf-0f695e5b805d_isrForCrosstalkSources_860802_11.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/69388045-56f7-4cb7-8ecf-0f695e5b805d_isrForCrosstalkSources_860802_11.stderr parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 326d7340e218c8770a5956c77615134b with result from task 18 parsl.dataflow.dflow DEBUG: Task 18 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/d2338ba0-7760-402f-8b22-d31cbbbc9285_isrForCrosstalkSources_860801_5.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/d2338ba0-7760-402f-8b22-d31cbbbc9285_isrForCrosstalkSources_860801_5.stderr parsl.dataflow.dflow INFO: Task 16 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 749f285c787b279f62d515d4b0c1139d with result from task 16 parsl.dataflow.dflow DEBUG: Task 16 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/9191625d-335c-4e77-b09d-5e0f115aca4f_isrForCrosstalkSources_860801_13.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/9191625d-335c-4e77-b09d-5e0f115aca4f_isrForCrosstalkSources_860801_13.stderr parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df8614665a08c9f9091ffb6f514feceb with result from task 36 parsl.dataflow.dflow DEBUG: Task 36 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/e1760c71-dcc5-469c-bded-8deb87623143_isrForCrosstalkSources_860801_44.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/e1760c71-dcc5-469c-bded-8deb87623143_isrForCrosstalkSources_860801_44.stderr parsl.dataflow.dflow INFO: Task 39 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ff538ec2f78275562891cbdb40f323f with result from task 39 parsl.dataflow.dflow DEBUG: Task 39 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/2abf3afa-98f6-4ccc-80b2-a3e0d3d81bfb_isrForCrosstalkSources_860802_18.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/2abf3afa-98f6-4ccc-80b2-a3e0d3d81bfb_isrForCrosstalkSources_860802_18.stderr parsl.dataflow.dflow INFO: Task 35 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c489046b58a961f7b6b7822e453ad50a with result from task 35 parsl.dataflow.dflow DEBUG: Task 35 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/c9ec1f95-cd5d-46b7-a5f2-36d80dd66872_isrForCrosstalkSources_860801_26.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/c9ec1f95-cd5d-46b7-a5f2-36d80dd66872_isrForCrosstalkSources_860801_26.stderr parsl.dataflow.dflow INFO: Task 37 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e887eb97406001014571832505b472d with result from task 37 parsl.dataflow.dflow DEBUG: Task 37 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/08668b3d-0289-4913-b965-6860481105c8_isrForCrosstalkSources_860803_45.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/08668b3d-0289-4913-b965-6860481105c8_isrForCrosstalkSources_860803_45.stderr parsl.dataflow.dflow INFO: Task 40 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 426dd8f023795c0d0df38ae46ff94918 with result from task 40 parsl.dataflow.dflow DEBUG: Task 40 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/51eb24ba-d807-42fb-a1ed-c2b3e0e9ac7b_isrForCrosstalkSources_860802_61.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/51eb24ba-d807-42fb-a1ed-c2b3e0e9ac7b_isrForCrosstalkSources_860802_61.stderr parsl.dataflow.dflow INFO: Task 34 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1caf9f5268c8dd823c1cb45ea803beed with result from task 34 parsl.dataflow.dflow DEBUG: Task 34 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/37f2f695-b247-48d1-8b33-76469229e1c2_isrForCrosstalkSources_860801_25.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/37f2f695-b247-48d1-8b33-76469229e1c2_isrForCrosstalkSources_860801_25.stderr parsl.dataflow.dflow INFO: Task 38 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d06e3c8c0e6cf045308223b7969d4fa with result from task 38 parsl.dataflow.dflow DEBUG: Task 38 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/015b04f1-e758-4003-8841-ebfb60cbe44c_isrForCrosstalkSources_860803_56.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/015b04f1-e758-4003-8841-ebfb60cbe44c_isrForCrosstalkSources_860803_56.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 138 parsl.dataflow.strategy DEBUG: Executor multi has 138 active tasks, 23/24 running/pending blocks, and 24 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23162860286032 parsl.dataflow.dflow INFO: Task 49 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c8d10dcc258a5a94ab1b541bdfa4cef with result from task 49 parsl.dataflow.dflow DEBUG: Task 49 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/ba95771b-5284-4cf5-b7b8-0c0108fd4ea8_isrForCrosstalkSources_860802_26.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/ba95771b-5284-4cf5-b7b8-0c0108fd4ea8_isrForCrosstalkSources_860802_26.stderr parsl.dataflow.dflow INFO: Task 52 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b91f4adfb471798ec11824cb0934ac3 with result from task 52 parsl.dataflow.dflow DEBUG: Task 52 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/0f68b69c-b2ad-4779-b81e-1b3182452cb8_isrForCrosstalkSources_860802_58.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/0f68b69c-b2ad-4779-b81e-1b3182452cb8_isrForCrosstalkSources_860802_58.stderr parsl.dataflow.dflow INFO: Task 50 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 636174db4a4a18da96147bcf1460fab7 with result from task 50 parsl.dataflow.dflow DEBUG: Task 50 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/d3e647e7-bd9e-48ee-aa6a-9f1af98422d9_isrForCrosstalkSources_860803_27.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/d3e647e7-bd9e-48ee-aa6a-9f1af98422d9_isrForCrosstalkSources_860803_27.stderr 2024-03-10 01:39:24 proc_lsst.multi:146 [INFO] found job 44401 in provider local proc_lsst.multi INFO: found job 44401 in provider local 2024-03-10 01:39:24 proc_lsst.multi:146 [INFO] found job 16971842 in provider astro proc_lsst.multi INFO: found job 16971842 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.dataflow.dflow INFO: Task 51 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5eea09734fc5b621ca89fc31fc2dab4 with result from task 51 parsl.dataflow.dflow DEBUG: Task 51 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/e1cb704e-52b5-44f7-acf1-405ac03c4a93_isrForCrosstalkSources_860803_52.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/e1cb704e-52b5-44f7-acf1-405ac03c4a93_isrForCrosstalkSources_860803_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:25 proc_lsst.multi:146 [INFO] found job 16971843 in provider astro proc_lsst.multi INFO: found job 16971843 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:25 proc_lsst.multi:146 [INFO] found job 16971844 in provider astro proc_lsst.multi INFO: found job 16971844 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.dataflow.dflow INFO: Task 53 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d059948c2ee3d619d006fa05ad46e08 with result from task 53 parsl.dataflow.dflow DEBUG: Task 53 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/3a4e24f1-abec-400b-92c4-336b26aec61f_isrForCrosstalkSources_860801_3.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/3a4e24f1-abec-400b-92c4-336b26aec61f_isrForCrosstalkSources_860801_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:26 proc_lsst.multi:146 [INFO] found job 16971845 in provider astro proc_lsst.multi INFO: found job 16971845 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:26 proc_lsst.multi:146 [INFO] found job 16971846 in provider astro proc_lsst.multi INFO: found job 16971846 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:27 proc_lsst.multi:146 [INFO] found job 16971847 in provider astro proc_lsst.multi INFO: found job 16971847 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:27 proc_lsst.multi:146 [INFO] found job 16971848 in provider astro proc_lsst.multi INFO: found job 16971848 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:28 proc_lsst.multi:146 [INFO] found job 16971849 in provider astro proc_lsst.multi INFO: found job 16971849 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:28 proc_lsst.multi:146 [INFO] found job 16971850 in provider astro proc_lsst.multi INFO: found job 16971850 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.dataflow.dflow INFO: Task 57 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21f5b0a13f5a48207243f69a65a8ecb7 with result from task 57 parsl.dataflow.dflow DEBUG: Task 57 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/02ee6012-555a-41ad-a6a0-718d5998d923_isrForCrosstalkSources_860802_17.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/02ee6012-555a-41ad-a6a0-718d5998d923_isrForCrosstalkSources_860802_17.stderr parsl.dataflow.dflow INFO: Task 58 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8436774457f828bc74fe31bba189d9c2 with result from task 58 parsl.dataflow.dflow DEBUG: Task 58 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/71f4b674-34ab-42da-899e-1a7ea487efa0_isrForCrosstalkSources_860803_43.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/71f4b674-34ab-42da-899e-1a7ea487efa0_isrForCrosstalkSources_860803_43.stderr parsl.dataflow.dflow INFO: Task 54 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f1ef8f7b2f60889a47d2002d9e4e3fb with result from task 54 parsl.dataflow.dflow DEBUG: Task 54 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/c09087a2-69ec-419a-8a4e-006e8e242185_isrForCrosstalkSources_860801_56.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/c09087a2-69ec-419a-8a4e-006e8e242185_isrForCrosstalkSources_860801_56.stderr parsl.dataflow.dflow INFO: Task 65 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80cf6ec34811760cc28525c76027a064 with result from task 65 parsl.dataflow.dflow DEBUG: Task 65 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/00d8ca68-90b2-4ef4-936e-3115ff835708_isrForCrosstalkSources_860802_51.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/00d8ca68-90b2-4ef4-936e-3115ff835708_isrForCrosstalkSources_860802_51.stderr parsl.dataflow.dflow INFO: Task 67 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35dbf53747cc5f1862d5a64e8b648731 with result from task 67 parsl.dataflow.dflow DEBUG: Task 67 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/540a11f8-6a8c-447c-ac52-8e6fbaf9a8ba_isrForCrosstalkSources_860802_39.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/540a11f8-6a8c-447c-ac52-8e6fbaf9a8ba_isrForCrosstalkSources_860802_39.stderr parsl.dataflow.dflow INFO: Task 71 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 153ecd9d60b9742f205be51a0e3dc2b0 with result from task 71 parsl.dataflow.dflow DEBUG: Task 71 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/1abc9cab-40ab-458f-9335-1eb658c4023b_isrForCrosstalkSources_860802_7.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/1abc9cab-40ab-458f-9335-1eb658c4023b_isrForCrosstalkSources_860802_7.stderr parsl.dataflow.dflow INFO: Task 69 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea5446c14026c25cff3bc1b6dbe380bc with result from task 69 parsl.dataflow.dflow DEBUG: Task 69 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/d36f88c2-844c-4ba5-a32f-3362e8c9d75a_isrForCrosstalkSources_860802_3.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/d36f88c2-844c-4ba5-a32f-3362e8c9d75a_isrForCrosstalkSources_860802_3.stderr parsl.dataflow.dflow INFO: Task 64 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry acbb09e2dfdb232abda8542a14255107 with result from task 64 parsl.dataflow.dflow DEBUG: Task 64 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/f9cab138-93a0-4750-a47a-925daf9eade6_isrForCrosstalkSources_860803_1.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/f9cab138-93a0-4750-a47a-925daf9eade6_isrForCrosstalkSources_860803_1.stderr parsl.dataflow.dflow INFO: Task 66 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d52eb850cba7696e6abb18376b35e189 with result from task 66 parsl.dataflow.dflow DEBUG: Task 66 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/35d9dc53-d6d6-4ed6-a775-4ed9ed9a59f0_isrForCrosstalkSources_860802_38.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/35d9dc53-d6d6-4ed6-a775-4ed9ed9a59f0_isrForCrosstalkSources_860802_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:31 proc_lsst.multi:146 [INFO] found job 16971851 in provider astro proc_lsst.multi INFO: found job 16971851 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.dataflow.dflow INFO: Task 63 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1a469704153eae1f8a0e07d9641cec4 with result from task 63 parsl.dataflow.dflow DEBUG: Task 63 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/5210ac67-faa0-497f-ab66-3749279c9297_isrForCrosstalkSources_860803_55.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/5210ac67-faa0-497f-ab66-3749279c9297_isrForCrosstalkSources_860803_55.stderr parsl.dataflow.dflow INFO: Task 68 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72db016853fcc9512cab1e1e8a3671ce with result from task 68 parsl.dataflow.dflow DEBUG: Task 68 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/e4d41443-9a67-4534-811a-0c2b1cdb36dc_isrForCrosstalkSources_860803_30.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/e4d41443-9a67-4534-811a-0c2b1cdb36dc_isrForCrosstalkSources_860803_30.stderr parsl.dataflow.dflow INFO: Task 62 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 986ea743e8646e55af009499ecd2df4e with result from task 62 parsl.dataflow.dflow DEBUG: Task 62 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/475c0ffc-b711-4acf-bab2-a17925ab3615_isrForCrosstalkSources_860803_53.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/475c0ffc-b711-4acf-bab2-a17925ab3615_isrForCrosstalkSources_860803_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:31 proc_lsst.multi:146 [INFO] found job 16971852 in provider astro proc_lsst.multi INFO: found job 16971852 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.dataflow.dflow INFO: Task 60 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1275aff259cfcdbac0bb323f3364852 with result from task 60 parsl.dataflow.dflow DEBUG: Task 60 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/edad5722-c057-44b8-9e9e-82ec2dd36e9a_isrForCrosstalkSources_860803_32.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/edad5722-c057-44b8-9e9e-82ec2dd36e9a_isrForCrosstalkSources_860803_32.stderr parsl.dataflow.dflow INFO: Task 70 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a41e451ea412ed4a67c800db8648dd7c with result from task 70 parsl.dataflow.dflow DEBUG: Task 70 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/f728410f-1518-4da8-b39b-8a820d5f3939_isrForCrosstalkSources_860801_60.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/f728410f-1518-4da8-b39b-8a820d5f3939_isrForCrosstalkSources_860801_60.stderr parsl.dataflow.dflow INFO: Task 59 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c04b75d9ba013f5b27b2dbd40058359 with result from task 59 parsl.dataflow.dflow DEBUG: Task 59 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/4f4cf0c8-7664-4016-a60d-37a89711efcb_isrForCrosstalkSources_860803_58.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/4f4cf0c8-7664-4016-a60d-37a89711efcb_isrForCrosstalkSources_860803_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:32 proc_lsst.multi:146 [INFO] found job 16971853 in provider astro proc_lsst.multi INFO: found job 16971853 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.dataflow.dflow INFO: Task 61 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b59b6831a4f50fea196b074df2bbefb8 with result from task 61 parsl.dataflow.dflow DEBUG: Task 61 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/73250313-55ba-43dc-95f3-23d79a30b5a3_isrForCrosstalkSources_860802_36.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/73250313-55ba-43dc-95f3-23d79a30b5a3_isrForCrosstalkSources_860802_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:32 proc_lsst.multi:146 [INFO] found job 16971854 in provider astro proc_lsst.multi INFO: found job 16971854 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:33 proc_lsst.multi:146 [INFO] found job 16971855 in provider astro proc_lsst.multi INFO: found job 16971855 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.dataflow.dflow INFO: Task 72 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f7b0c8194e803995b122642b021cada with result from task 72 parsl.dataflow.dflow DEBUG: Task 72 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/ba94904a-2d1f-4ea3-99d8-ec9f12987d7a_isrForCrosstalkSources_860802_56.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/ba94904a-2d1f-4ea3-99d8-ec9f12987d7a_isrForCrosstalkSources_860802_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:33 proc_lsst.multi:146 [INFO] found job 16971856 in provider astro proc_lsst.multi INFO: found job 16971856 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:34 proc_lsst.multi:146 [INFO] found job 16971857 in provider astro proc_lsst.multi INFO: found job 16971857 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.dataflow.dflow INFO: Task 55 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a992622ffcef472317f7479af199a23 with result from task 55 parsl.dataflow.dflow DEBUG: Task 55 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/d62e3bea-8f52-4554-8a28-9b7e6e5659b6_isrForCrosstalkSources_860803_48.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/d62e3bea-8f52-4554-8a28-9b7e6e5659b6_isrForCrosstalkSources_860803_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:34 proc_lsst.multi:146 [INFO] found job 16971858 in provider astro proc_lsst.multi INFO: found job 16971858 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.dataflow.dflow INFO: Task 56 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90a38bdd8ea01ba81d3b58c645e85c46 with result from task 56 parsl.dataflow.dflow DEBUG: Task 56 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/d0f3281b-b9cb-46bc-adff-72180a402c77_isrForCrosstalkSources_860801_54.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/d0f3281b-b9cb-46bc-adff-72180a402c77_isrForCrosstalkSources_860801_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:35 proc_lsst.multi:146 [INFO] found job 16971859 in provider astro proc_lsst.multi INFO: found job 16971859 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:36 proc_lsst.multi:146 [INFO] found job 16971860 in provider astro proc_lsst.multi INFO: found job 16971860 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:36 proc_lsst.multi:146 [INFO] found job 16971861 in provider astro proc_lsst.multi INFO: found job 16971861 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.dataflow.dflow INFO: Task 76 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dab9a89080f1521f97627e3dc0dc86ad with result from task 76 parsl.dataflow.dflow DEBUG: Task 76 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/a591e198-d484-4b77-90ca-dd12e6fa44ea_isrForCrosstalkSources_860803_24.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/a591e198-d484-4b77-90ca-dd12e6fa44ea_isrForCrosstalkSources_860803_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:37 proc_lsst.multi:146 [INFO] found job 16971862 in provider astro proc_lsst.multi INFO: found job 16971862 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.dataflow.dflow INFO: Task 73 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00f7d8f9266f92652827fb3f87ad880d with result from task 73 parsl.dataflow.dflow DEBUG: Task 73 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/4de187bc-91c9-41a7-b14e-6a7920522376_isrForCrosstalkSources_860803_35.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/4de187bc-91c9-41a7-b14e-6a7920522376_isrForCrosstalkSources_860803_35.stderr parsl.dataflow.dflow INFO: Task 74 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e91bd8fc23a1132b39e109c681de14d with result from task 74 parsl.dataflow.dflow DEBUG: Task 74 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/e4ef14de-2d95-4904-8959-a07001dbb070_isrForCrosstalkSources_860802_44.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/e4ef14de-2d95-4904-8959-a07001dbb070_isrForCrosstalkSources_860802_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:37 proc_lsst.multi:146 [INFO] found job 16971863 in provider astro proc_lsst.multi INFO: found job 16971863 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:38 proc_lsst.multi:146 [INFO] found job 16971864 in provider astro proc_lsst.multi INFO: found job 16971864 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.dataflow.dflow INFO: Task 75 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63de4c4edbc4eee5f547f829c30d0d9b with result from task 75 parsl.dataflow.dflow DEBUG: Task 75 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/976c5646-4ecf-45d4-8232-ca10ac21b804_isrForCrosstalkSources_860803_13.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/976c5646-4ecf-45d4-8232-ca10ac21b804_isrForCrosstalkSources_860803_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:38 proc_lsst.multi:146 [INFO] found job 16971865 in provider astro proc_lsst.multi INFO: found job 16971865 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.dataflow.dflow INFO: Task 77 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ef733eab4c57ad44e9b8e05c84c6273 with result from task 77 parsl.dataflow.dflow DEBUG: Task 77 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/25c7332f-73f0-4b0c-b164-cc85a3888a5f_isrForCrosstalkSources_860801_46.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/25c7332f-73f0-4b0c-b164-cc85a3888a5f_isrForCrosstalkSources_860801_46.stderr parsl.dataflow.dflow INFO: Task 78 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5175a34aa65edf5a3a459387e02c9cce with result from task 78 parsl.dataflow.dflow DEBUG: Task 78 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/747a3582-ba4c-4b69-b097-4e00084f8563_isrForCrosstalkSources_860801_24.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/747a3582-ba4c-4b69-b097-4e00084f8563_isrForCrosstalkSources_860801_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:39 proc_lsst.multi:146 [INFO] found job 16971866 in provider astro proc_lsst.multi INFO: found job 16971866 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:40 proc_lsst.multi:146 [INFO] found job 16971867 in provider astro proc_lsst.multi INFO: found job 16971867 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:40 proc_lsst.multi:146 [INFO] found job 16971868 in provider astro proc_lsst.multi INFO: found job 16971868 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:41 proc_lsst.multi:146 [INFO] found job 16971869 in provider astro proc_lsst.multi INFO: found job 16971869 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:41 proc_lsst.multi:146 [INFO] found job 16971870 in provider astro proc_lsst.multi INFO: found job 16971870 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:42 proc_lsst.multi:146 [INFO] found job 16971871 in provider astro proc_lsst.multi INFO: found job 16971871 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.dataflow.dflow INFO: Task 45 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d66429f7cafc7a81463373518cfb299 with result from task 45 parsl.dataflow.dflow DEBUG: Task 45 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/c6e088e1-25db-49e8-8cc3-4861acc12d0b_isrForCrosstalkSources_860801_42.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/c6e088e1-25db-49e8-8cc3-4861acc12d0b_isrForCrosstalkSources_860801_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:42 proc_lsst.multi:146 [INFO] found job 16971872 in provider escience_rtx6k proc_lsst.multi INFO: found job 16971872 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971872,16971873,16971874,16971875,16971877' parsl.dataflow.dflow INFO: Task 79 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d835a1b1fded08525b33efa35f7deea7 with result from task 79 parsl.dataflow.dflow DEBUG: Task 79 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/c9bf7867-3e01-4519-8f26-17774e809fd4_isrForCrosstalkSources_860803_23.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/c9bf7867-3e01-4519-8f26-17774e809fd4_isrForCrosstalkSources_860803_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971877 R 16971872 R 16971873 R 16971874 R 16971875 R parsl.providers.slurm.slurm DEBUG: Updating job 16971877 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971872 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971873 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971874 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971875 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:43 proc_lsst.multi:146 [INFO] found job 16971873 in provider escience_rtx6k proc_lsst.multi INFO: found job 16971873 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971872,16971873,16971874,16971875,16971877' parsl.providers.slurm.slurm DEBUG: squeue returned 16971877 R 16971872 R 16971873 R 16971874 R 16971875 R parsl.providers.slurm.slurm DEBUG: Updating job 16971877 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971872 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971873 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971874 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971875 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:43 proc_lsst.multi:146 [INFO] found job 16971874 in provider escience_rtx6k proc_lsst.multi INFO: found job 16971874 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971872,16971873,16971874,16971875,16971877' parsl.providers.slurm.slurm DEBUG: squeue returned 16971877 R 16971872 R 16971873 R 16971874 R 16971875 R parsl.providers.slurm.slurm DEBUG: Updating job 16971877 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971872 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971873 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971874 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971875 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:44 proc_lsst.multi:146 [INFO] found job 16971875 in provider escience_rtx6k proc_lsst.multi INFO: found job 16971875 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971872,16971873,16971874,16971875,16971877' parsl.dataflow.dflow INFO: Task 86 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49ffacff01996791cbe6555775da7271 with result from task 86 parsl.dataflow.dflow DEBUG: Task 86 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/f86023d0-0cd7-429a-aa7e-7052a3fe21cf_isrForCrosstalkSources_860801_52.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/f86023d0-0cd7-429a-aa7e-7052a3fe21cf_isrForCrosstalkSources_860801_52.stderr parsl.dataflow.dflow INFO: Task 90 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c244e37d6e2e996626cf01d657abf113 with result from task 90 parsl.dataflow.dflow DEBUG: Task 90 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/87e135f3-b1b5-4365-836a-c0d7668bd9a6_isrForCrosstalkSources_860801_34.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/87e135f3-b1b5-4365-836a-c0d7668bd9a6_isrForCrosstalkSources_860801_34.stderr parsl.dataflow.dflow INFO: Task 87 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a835c8b96d3c822a8a23a27d8da06bb4 with result from task 87 parsl.dataflow.dflow DEBUG: Task 87 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/00581f0c-8ec6-4398-bd41-2eea2db71020_isrForCrosstalkSources_860801_22.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/00581f0c-8ec6-4398-bd41-2eea2db71020_isrForCrosstalkSources_860801_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971877 R 16971872 R 16971873 R 16971874 R 16971875 R parsl.providers.slurm.slurm DEBUG: Updating job 16971877 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971872 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971873 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971874 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971875 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:44 proc_lsst.multi:146 [INFO] found job 16971877 in provider escience_rtx6k proc_lsst.multi INFO: found job 16971877 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971872,16971873,16971874,16971875,16971877' parsl.dataflow.dflow INFO: Task 83 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 779f74dff7aae9c7cdbd802dad4161a6 with result from task 83 parsl.dataflow.dflow DEBUG: Task 83 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/5110fffe-c6ef-4eb3-a658-aebb188e8f2a_isrForCrosstalkSources_860801_58.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/5110fffe-c6ef-4eb3-a658-aebb188e8f2a_isrForCrosstalkSources_860801_58.stderr parsl.dataflow.dflow INFO: Task 88 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a33605eb0278296d2c6a42956820d76 with result from task 88 parsl.dataflow.dflow DEBUG: Task 88 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/b5bdb31a-f78e-44a1-b450-a4d6f18e31ad_isrForCrosstalkSources_860803_3.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/b5bdb31a-f78e-44a1-b450-a4d6f18e31ad_isrForCrosstalkSources_860803_3.stderr parsl.dataflow.dflow INFO: Task 85 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b361236fc075e53f2f458f0d0fac7f6 with result from task 85 parsl.dataflow.dflow DEBUG: Task 85 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/9ba5bd30-8427-442f-b459-848edd93673e_isrForCrosstalkSources_860803_20.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/9ba5bd30-8427-442f-b459-848edd93673e_isrForCrosstalkSources_860803_20.stderr parsl.dataflow.dflow INFO: Task 84 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f1cfd3a8c2d792d3ba72271f9c6048c6 with result from task 84 parsl.dataflow.dflow DEBUG: Task 84 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/51cebf7f-e54d-4f04-8cc5-20b30f4c94a6_isrForCrosstalkSources_860802_8.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/51cebf7f-e54d-4f04-8cc5-20b30f4c94a6_isrForCrosstalkSources_860802_8.stderr parsl.dataflow.dflow INFO: Task 89 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe440ee5e00bc697c763cfe6b545cbbc with result from task 89 parsl.dataflow.dflow DEBUG: Task 89 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/7963aea7-cd1c-4258-aa4c-8e100a1f800c_isrForCrosstalkSources_860801_38.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/7963aea7-cd1c-4258-aa4c-8e100a1f800c_isrForCrosstalkSources_860801_38.stderr parsl.dataflow.dflow INFO: Task 92 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 542bba6ae6449fc7d39a29955f3ef4c9 with result from task 92 parsl.dataflow.dflow DEBUG: Task 92 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/fc6b128a-0d4c-4b2a-b3f1-ca07ae121d63_isrForCrosstalkSources_860801_18.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/fc6b128a-0d4c-4b2a-b3f1-ca07ae121d63_isrForCrosstalkSources_860801_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971877 R 16971872 R 16971873 R 16971874 R 16971875 R parsl.providers.slurm.slurm DEBUG: Updating job 16971877 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971872 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971873 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971874 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971875 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:45 proc_lsst.multi:146 [INFO] found job 16971878 in provider escience_a40 proc_lsst.multi INFO: found job 16971878 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.dataflow.dflow INFO: Task 80 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 860126842f66adbaa87ed4889061a242 with result from task 80 parsl.dataflow.dflow DEBUG: Task 80 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/2a1d8f9b-460c-4cb7-b552-ff9ef7e9450b_isrForCrosstalkSources_860802_29.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/2a1d8f9b-460c-4cb7-b552-ff9ef7e9450b_isrForCrosstalkSources_860802_29.stderr parsl.dataflow.dflow INFO: Task 81 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0408e81154f5d9a62d76bed298c6d4f0 with result from task 81 parsl.dataflow.dflow DEBUG: Task 81 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/809d64af-9464-4f4c-b0eb-c2f6b0fbf44f_isrForCrosstalkSources_860802_50.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/809d64af-9464-4f4c-b0eb-c2f6b0fbf44f_isrForCrosstalkSources_860802_50.stderr parsl.dataflow.dflow INFO: Task 82 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09950ff524e755c81f55c2a2489babf0 with result from task 82 parsl.dataflow.dflow DEBUG: Task 82 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/4d3f496e-a0a0-4aa8-a981-e67b8fcee380_isrForCrosstalkSources_860802_43.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/4d3f496e-a0a0-4aa8-a981-e67b8fcee380_isrForCrosstalkSources_860802_43.stderr parsl.dataflow.dflow INFO: Task 91 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 178f6aa571546be93d55cf135630d8ed with result from task 91 parsl.dataflow.dflow DEBUG: Task 91 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/a3a27500-75b6-47b3-b16e-1b189d55af2f_isrForCrosstalkSources_860802_14.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/a3a27500-75b6-47b3-b16e-1b189d55af2f_isrForCrosstalkSources_860802_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:45 proc_lsst.multi:146 [INFO] found job 16971879 in provider escience_a40 proc_lsst.multi INFO: found job 16971879 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:46 proc_lsst.multi:146 [INFO] found job 16971880 in provider escience_a40 proc_lsst.multi INFO: found job 16971880 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.dataflow.dflow INFO: Task 93 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77b15fcaa17574ad2bcdd0ee90c92b22 with result from task 93 parsl.dataflow.dflow DEBUG: Task 93 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/b828c7b7-5ca3-4385-a7ba-f55f0066b29d_isrForCrosstalkSources_860801_35.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/b828c7b7-5ca3-4385-a7ba-f55f0066b29d_isrForCrosstalkSources_860801_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:46 proc_lsst.multi:146 [INFO] found job 16971881 in provider escience_a40 proc_lsst.multi INFO: found job 16971881 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:47 proc_lsst.multi:146 [INFO] found job 16971882 in provider escience_a40 proc_lsst.multi INFO: found job 16971882 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.dataflow.dflow INFO: Task 104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 572e468d95955f7e5ac52cff31c23cf3 with result from task 104 parsl.dataflow.dflow DEBUG: Task 104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/21f0cf9f-409e-45cb-bffa-3c2f694704b0_isrForCrosstalkSources_860802_4.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/21f0cf9f-409e-45cb-bffa-3c2f694704b0_isrForCrosstalkSources_860802_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:47 proc_lsst.multi:146 [INFO] found job 16971883 in provider escience_a40 proc_lsst.multi INFO: found job 16971883 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.dataflow.dflow INFO: Task 100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8c583f29c3fc7089f5225e4cb675297 with result from task 100 parsl.dataflow.dflow DEBUG: Task 100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/21359ac5-ac6b-43c2-aa01-cff16b1e12e6_isrForCrosstalkSources_860801_6.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/21359ac5-ac6b-43c2-aa01-cff16b1e12e6_isrForCrosstalkSources_860801_6.stderr parsl.dataflow.dflow INFO: Task 105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bcc00d3368f604f97b70a50c0bcea2c3 with result from task 105 parsl.dataflow.dflow DEBUG: Task 105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/69f54418-1cb6-4e1c-8aee-380b22c75a21_isrForCrosstalkSources_860802_59.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/69f54418-1cb6-4e1c-8aee-380b22c75a21_isrForCrosstalkSources_860802_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:48 proc_lsst.multi:146 [INFO] found job 16971884 in provider escience_a40 proc_lsst.multi INFO: found job 16971884 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.dataflow.dflow INFO: Task 98 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f38f6c0038fb02e20585e6b813025b3 with result from task 98 parsl.dataflow.dflow DEBUG: Task 98 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/2517a6aa-3dce-47c0-aa9d-b6565c5494f0_isrForCrosstalkSources_860803_16.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/2517a6aa-3dce-47c0-aa9d-b6565c5494f0_isrForCrosstalkSources_860803_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:48 proc_lsst.multi:146 [INFO] found job 16971885 in provider escience_a40 proc_lsst.multi INFO: found job 16971885 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.dataflow.dflow INFO: Task 101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0518c6d10fcc0430f6df3a133f21071 with result from task 101 parsl.dataflow.dflow DEBUG: Task 101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/ddc3b331-d93e-40a9-bcde-9594d7c6f587_isrForCrosstalkSources_860803_40.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/ddc3b331-d93e-40a9-bcde-9594d7c6f587_isrForCrosstalkSources_860803_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:49 proc_lsst.multi:146 [INFO] found job 16971886 in provider escience_a40 proc_lsst.multi INFO: found job 16971886 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.dataflow.dflow INFO: Task 99 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb520e4e154a41730c9210e4d4fea901 with result from task 99 parsl.dataflow.dflow DEBUG: Task 99 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/9b0c12a2-b2b0-453d-be5b-9549d6e2baee_isrForCrosstalkSources_860803_6.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/9b0c12a2-b2b0-453d-be5b-9549d6e2baee_isrForCrosstalkSources_860803_6.stderr parsl.dataflow.dflow INFO: Task 102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a089add211b929bf5d36a2c77ac7e091 with result from task 102 parsl.dataflow.dflow DEBUG: Task 102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/49999342-acc3-483a-9d11-367d7688d326_isrForCrosstalkSources_860803_29.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/49999342-acc3-483a-9d11-367d7688d326_isrForCrosstalkSources_860803_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:49 proc_lsst.multi:146 [INFO] found job 16971887 in provider escience_a40 proc_lsst.multi INFO: found job 16971887 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:50 proc_lsst.multi:146 [INFO] found job 16971888 in provider escience_a40 proc_lsst.multi INFO: found job 16971888 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.dataflow.dflow INFO: Task 103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ac35f8b106cfb3679c8f3847792315f with result from task 103 parsl.dataflow.dflow DEBUG: Task 103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/0519d253-9b16-4f97-9d1a-679632264654_isrForCrosstalkSources_860801_4.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/0519d253-9b16-4f97-9d1a-679632264654_isrForCrosstalkSources_860801_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 84 parsl.dataflow.strategy DEBUG: Executor multi has 84 active tasks, 47/0 running/pending blocks, and 28 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23162860286032 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 84 parsl.dataflow.strategy DEBUG: Executor multi has 84 active tasks, 47/0 running/pending blocks, and 28 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23162860286032 parsl.dataflow.dflow INFO: Task 112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f37cc51b67dd5d9dec6d3a12cafbf22 with result from task 112 parsl.dataflow.dflow DEBUG: Task 112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/d179c47e-e6e7-415b-bcd5-e826fefc6629_isrForCrosstalkSources_860803_18.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/d179c47e-e6e7-415b-bcd5-e826fefc6629_isrForCrosstalkSources_860803_18.stderr parsl.dataflow.dflow INFO: Task 117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f3874d712b07f06379cf522d600a3f5 with result from task 117 parsl.dataflow.dflow DEBUG: Task 117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/15be0586-0605-4322-bc0c-facc3682322c_isrForCrosstalkSources_860802_24.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/15be0586-0605-4322-bc0c-facc3682322c_isrForCrosstalkSources_860802_24.stderr parsl.dataflow.dflow INFO: Task 114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf55ac4e2b772de289bfb8cfd5aea4e2 with result from task 114 parsl.dataflow.dflow DEBUG: Task 114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/833a3654-d118-4003-b64a-5ce6de0b06ff_isrForCrosstalkSources_860802_25.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/833a3654-d118-4003-b64a-5ce6de0b06ff_isrForCrosstalkSources_860802_25.stderr parsl.dataflow.dflow INFO: Task 108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e726d5991d4f86c34ec619cf9a3113f with result from task 108 parsl.dataflow.dflow DEBUG: Task 108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/54e1b909-9b32-444b-ac0c-6d0fe2b0f0bc_isrForCrosstalkSources_860801_28.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/54e1b909-9b32-444b-ac0c-6d0fe2b0f0bc_isrForCrosstalkSources_860801_28.stderr parsl.dataflow.dflow INFO: Task 113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 747424205dda1dd6aa9a4925c87882f8 with result from task 113 parsl.dataflow.dflow DEBUG: Task 113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/bc385d76-403e-423e-ac51-aecafda90317_isrForCrosstalkSources_860801_8.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/bc385d76-403e-423e-ac51-aecafda90317_isrForCrosstalkSources_860801_8.stderr parsl.dataflow.dflow INFO: Task 118 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 129dcfa7b50cf1bd68593dbd6f8351a3 with result from task 118 parsl.dataflow.dflow DEBUG: Task 118 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/cc4ca081-9b18-417f-b625-09f695f9d91b_isrForCrosstalkSources_860801_40.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/cc4ca081-9b18-417f-b625-09f695f9d91b_isrForCrosstalkSources_860801_40.stderr parsl.dataflow.dflow INFO: Task 109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa7920596d5e0a4d78f794553eea1467 with result from task 109 parsl.dataflow.dflow DEBUG: Task 109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/a01f995c-e6a8-4ea4-9822-4cbc18f0bde4_isrForCrosstalkSources_860801_9.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/a01f995c-e6a8-4ea4-9822-4cbc18f0bde4_isrForCrosstalkSources_860801_9.stderr parsl.dataflow.dflow INFO: Task 111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3578c5bb198229b55878c13731b6323c with result from task 111 parsl.dataflow.dflow DEBUG: Task 111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/eb99815a-05ad-4a4c-a587-349ca1b97c98_isrForCrosstalkSources_860801_53.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/eb99815a-05ad-4a4c-a587-349ca1b97c98_isrForCrosstalkSources_860801_53.stderr parsl.dataflow.dflow INFO: Task 107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef32ffbf71807e6a86f3c33154315ec2 with result from task 107 parsl.dataflow.dflow DEBUG: Task 107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/d7a95ccf-22a8-4534-854b-b0d9d8281752_isrForCrosstalkSources_860803_28.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/d7a95ccf-22a8-4534-854b-b0d9d8281752_isrForCrosstalkSources_860803_28.stderr parsl.dataflow.dflow INFO: Task 110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c10ee34403a681d27e78ad62492e2b83 with result from task 110 parsl.dataflow.dflow DEBUG: Task 110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/87e66c6c-c7c9-413a-9984-aed53b61173f_isrForCrosstalkSources_860803_57.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/87e66c6c-c7c9-413a-9984-aed53b61173f_isrForCrosstalkSources_860803_57.stderr parsl.dataflow.dflow INFO: Task 119 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0dabe1770d40dd930fec3f387118009d with result from task 119 parsl.dataflow.dflow DEBUG: Task 119 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/166eb073-2976-422b-8d8d-fc13b973b3e9_isrForCrosstalkSources_860803_62.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/166eb073-2976-422b-8d8d-fc13b973b3e9_isrForCrosstalkSources_860803_62.stderr 2024-03-10 01:39:56 proc_lsst.multi:146 [INFO] found job 44401 in provider local proc_lsst.multi INFO: found job 44401 in provider local 2024-03-10 01:39:56 proc_lsst.multi:146 [INFO] found job 16971842 in provider astro proc_lsst.multi INFO: found job 16971842 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:56 proc_lsst.multi:146 [INFO] found job 16971843 in provider astro proc_lsst.multi INFO: found job 16971843 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.dataflow.dflow INFO: Task 122 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b2f5975e0f8a4cc3438e72d1fb53691 with result from task 122 parsl.dataflow.dflow DEBUG: Task 122 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/e3dab589-8988-404b-8386-08b9c179e8bb_isrForCrosstalkSources_860801_50.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/e3dab589-8988-404b-8386-08b9c179e8bb_isrForCrosstalkSources_860801_50.stderr parsl.dataflow.dflow INFO: Task 124 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 980c7a8c88be4f1a3b1650343078a406 with result from task 124 parsl.dataflow.dflow DEBUG: Task 124 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/bd47f21a-1814-480a-98c4-5ab5b2011ad5_isrForCrosstalkSources_860803_59.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/bd47f21a-1814-480a-98c4-5ab5b2011ad5_isrForCrosstalkSources_860803_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:57 proc_lsst.multi:146 [INFO] found job 16971844 in provider astro proc_lsst.multi INFO: found job 16971844 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:57 proc_lsst.multi:146 [INFO] found job 16971845 in provider astro proc_lsst.multi INFO: found job 16971845 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.dataflow.dflow INFO: Task 116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4cf5b2b26ed3a3696a3649054cfd4f1 with result from task 116 parsl.dataflow.dflow DEBUG: Task 116 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/61332636-d9c2-482c-b203-fd7ec6c37fc2_isrForCrosstalkSources_860802_10.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/61332636-d9c2-482c-b203-fd7ec6c37fc2_isrForCrosstalkSources_860802_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:58 proc_lsst.multi:146 [INFO] found job 16971846 in provider astro proc_lsst.multi INFO: found job 16971846 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.dataflow.dflow INFO: Task 120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2691a799c7a544599c32eb2333c799d2 with result from task 120 parsl.dataflow.dflow DEBUG: Task 120 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/0856ee1c-1318-4375-a763-820f2423dd9e_isrForCrosstalkSources_860803_2.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/0856ee1c-1318-4375-a763-820f2423dd9e_isrForCrosstalkSources_860803_2.stderr parsl.dataflow.dflow INFO: Task 115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbbe1b83a6f06a26402a9a3c75d8f1bf with result from task 115 parsl.dataflow.dflow DEBUG: Task 115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/e0ff9c89-b24c-4c7e-9332-6dc407000d12_isrForCrosstalkSources_860803_60.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/e0ff9c89-b24c-4c7e-9332-6dc407000d12_isrForCrosstalkSources_860803_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:58 proc_lsst.multi:146 [INFO] found job 16971847 in provider astro proc_lsst.multi INFO: found job 16971847 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.dataflow.dflow INFO: Task 121 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49faa0b102e2b5222d28348353acea8d with result from task 121 parsl.dataflow.dflow DEBUG: Task 121 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/804d7d39-346a-4b15-a35b-501c9b1b7881_isrForCrosstalkSources_860802_49.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/804d7d39-346a-4b15-a35b-501c9b1b7881_isrForCrosstalkSources_860802_49.stderr parsl.dataflow.dflow INFO: Task 123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 932ded8fa6fc8a87775df2c94686b80b with result from task 123 parsl.dataflow.dflow DEBUG: Task 123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/a6fa9773-9d34-4105-8986-06c342035102_isrForCrosstalkSources_860803_37.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/a6fa9773-9d34-4105-8986-06c342035102_isrForCrosstalkSources_860803_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:59 proc_lsst.multi:146 [INFO] found job 16971848 in provider astro proc_lsst.multi INFO: found job 16971848 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:39:59 proc_lsst.multi:146 [INFO] found job 16971849 in provider astro proc_lsst.multi INFO: found job 16971849 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:00 proc_lsst.multi:146 [INFO] found job 16971850 in provider astro proc_lsst.multi INFO: found job 16971850 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:01 proc_lsst.multi:146 [INFO] found job 16971851 in provider astro proc_lsst.multi INFO: found job 16971851 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.dataflow.dflow INFO: Task 128 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 995d997c2fb469b4656b8576f25b9f8b with result from task 128 parsl.dataflow.dflow DEBUG: Task 128 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/de11ad99-eff4-4436-a9c7-10b0c253e178_isrForCrosstalkSources_860802_22.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/de11ad99-eff4-4436-a9c7-10b0c253e178_isrForCrosstalkSources_860802_22.stderr parsl.dataflow.dflow INFO: Task 126 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 065ef561e89805827a0477062b8a051b with result from task 126 parsl.dataflow.dflow DEBUG: Task 126 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/77663be1-bd68-4e79-85d4-7299414fff75_isrForCrosstalkSources_860801_48.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/77663be1-bd68-4e79-85d4-7299414fff75_isrForCrosstalkSources_860801_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:03 proc_lsst.multi:146 [INFO] found job 16971852 in provider astro proc_lsst.multi INFO: found job 16971852 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.dataflow.dflow INFO: Task 129 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ea14aeaf9a78d44153682535569cc3b with result from task 129 parsl.dataflow.dflow DEBUG: Task 129 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/c563faf9-eee6-4cb8-bc60-7141ae803d72_isrForCrosstalkSources_860803_10.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/c563faf9-eee6-4cb8-bc60-7141ae803d72_isrForCrosstalkSources_860803_10.stderr parsl.dataflow.dflow INFO: Task 134 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b83311e6ba24f3e3043274506f16b06 with result from task 134 parsl.dataflow.dflow DEBUG: Task 134 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/9f6071e7-22c4-4252-aabe-a8f738cc7abb_isrForCrosstalkSources_860801_32.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/9f6071e7-22c4-4252-aabe-a8f738cc7abb_isrForCrosstalkSources_860801_32.stderr parsl.dataflow.dflow INFO: Task 137 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4ff91a6ae55e487e0ff8ffd692cadaa with result from task 137 parsl.dataflow.dflow DEBUG: Task 137 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/4326038d-5407-4345-a3c8-ea7aeceebd3d_isrForCrosstalkSources_860801_14.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/4326038d-5407-4345-a3c8-ea7aeceebd3d_isrForCrosstalkSources_860801_14.stderr parsl.dataflow.dflow INFO: Task 125 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8bae0e849bb7c4b21d14f22648f43b18 with result from task 125 parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:04 proc_lsst.multi:146 [INFO] found job 16971853 in provider astro proc_lsst.multi INFO: found job 16971853 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.dataflow.dflow DEBUG: Task 125 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/59a7407b-5004-4144-9b0f-72f09780ed21_isrForCrosstalkSources_860803_41.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/59a7407b-5004-4144-9b0f-72f09780ed21_isrForCrosstalkSources_860803_41.stderr parsl.dataflow.dflow INFO: Task 140 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ecbb98d85a7c6036ca72e39d63772a4 with result from task 140 parsl.dataflow.dflow DEBUG: Task 140 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/994650d7-3232-4985-8b3c-b3b001b50931_isrForCrosstalkSources_860801_29.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/994650d7-3232-4985-8b3c-b3b001b50931_isrForCrosstalkSources_860801_29.stderr parsl.dataflow.dflow INFO: Task 131 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 630ac7f260f710de82d8e25d29b27d97 with result from task 131 parsl.dataflow.dflow DEBUG: Task 131 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/7160d22e-bfb8-4ba4-a69c-1d08faba322b_isrForCrosstalkSources_860801_59.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/7160d22e-bfb8-4ba4-a69c-1d08faba322b_isrForCrosstalkSources_860801_59.stderr parsl.dataflow.dflow INFO: Task 136 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0b25036656cedd27aac0d1be281f79e with result from task 136 parsl.dataflow.dflow DEBUG: Task 136 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/126de834-1baf-4fcc-ae27-6a4baebd8911_isrForCrosstalkSources_860803_25.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/126de834-1baf-4fcc-ae27-6a4baebd8911_isrForCrosstalkSources_860803_25.stderr parsl.dataflow.dflow INFO: Task 130 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aed4684df6b82653694e7a34dbcb53c8 with result from task 130 parsl.dataflow.dflow DEBUG: Task 130 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/dc885483-6018-4b01-b102-59b7e2d7f926_isrForCrosstalkSources_860802_30.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/dc885483-6018-4b01-b102-59b7e2d7f926_isrForCrosstalkSources_860802_30.stderr parsl.dataflow.dflow INFO: Task 132 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa3db82dc4eb76b61ec597f2a5e038e0 with result from task 132 parsl.dataflow.dflow DEBUG: Task 132 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/d0f9db30-118c-40cc-af8a-ec84f1efbad3_isrForCrosstalkSources_860802_53.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/d0f9db30-118c-40cc-af8a-ec84f1efbad3_isrForCrosstalkSources_860802_53.stderr parsl.dataflow.dflow INFO: Task 139 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01af022b7a11ed81e84476affa933691 with result from task 139 parsl.dataflow.dflow DEBUG: Task 139 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/beb6495b-922f-42af-87ab-249cc3bb3fd5_isrForCrosstalkSources_860802_54.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/beb6495b-922f-42af-87ab-249cc3bb3fd5_isrForCrosstalkSources_860802_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:04 proc_lsst.multi:146 [INFO] found job 16971854 in provider astro proc_lsst.multi INFO: found job 16971854 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:05 proc_lsst.multi:146 [INFO] found job 16971855 in provider astro proc_lsst.multi INFO: found job 16971855 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.dataflow.dflow INFO: Task 127 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 950d54daafe597a22228ebc5b5396bf6 with result from task 127 parsl.dataflow.dflow DEBUG: Task 127 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/98d87c94-f6c0-4adb-b155-b3896b186e61_isrForCrosstalkSources_860801_10.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/98d87c94-f6c0-4adb-b155-b3896b186e61_isrForCrosstalkSources_860801_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:05 proc_lsst.multi:146 [INFO] found job 16971856 in provider astro proc_lsst.multi INFO: found job 16971856 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.dataflow.dflow INFO: Task 133 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry baadb6b43e9769a6e169464c6ed6980a with result from task 133 parsl.dataflow.dflow DEBUG: Task 133 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/71a1ae0d-7c1f-4013-aa88-b18aadba12b9_isrForCrosstalkSources_860803_33.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/71a1ae0d-7c1f-4013-aa88-b18aadba12b9_isrForCrosstalkSources_860803_33.stderr parsl.dataflow.dflow INFO: Task 135 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2b0d41a74a9296e99c3afa45b4abe95 with result from task 135 parsl.dataflow.dflow DEBUG: Task 135 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/83a3c6a4-493c-4021-b89b-af981dbe904c_isrForCrosstalkSources_860802_5.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/83a3c6a4-493c-4021-b89b-af981dbe904c_isrForCrosstalkSources_860802_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:06 proc_lsst.multi:146 [INFO] found job 16971857 in provider astro proc_lsst.multi INFO: found job 16971857 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.dataflow.dflow INFO: Task 141 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5b4c015c02befccc8357b51a580636a with result from task 141 parsl.dataflow.dflow DEBUG: Task 141 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/59eca0b7-5253-4fc8-a4b1-77acbc256128_isrForCrosstalkSources_860801_2.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/59eca0b7-5253-4fc8-a4b1-77acbc256128_isrForCrosstalkSources_860801_2.stderr parsl.dataflow.dflow INFO: Task 142 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4132e504bc1a88ac31cfc3a66b6023c9 with result from task 142 parsl.dataflow.dflow DEBUG: Task 142 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/181a830c-9ef4-4a52-bb78-2b106c9c08e9_isrForCrosstalkSources_860801_55.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/181a830c-9ef4-4a52-bb78-2b106c9c08e9_isrForCrosstalkSources_860801_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:06 proc_lsst.multi:146 [INFO] found job 16971858 in provider astro proc_lsst.multi INFO: found job 16971858 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:07 proc_lsst.multi:146 [INFO] found job 16971859 in provider astro proc_lsst.multi INFO: found job 16971859 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:07 proc_lsst.multi:146 [INFO] found job 16971860 in provider astro proc_lsst.multi INFO: found job 16971860 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.dataflow.dflow INFO: Task 138 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa1f11f61089ee79319c49bc9be7d3c0 with result from task 138 parsl.dataflow.dflow DEBUG: Task 138 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/722dd519-c143-475b-a818-fd7d0407d886_isrForCrosstalkSources_860803_46.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/722dd519-c143-475b-a818-fd7d0407d886_isrForCrosstalkSources_860803_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:08 proc_lsst.multi:146 [INFO] found job 16971861 in provider astro proc_lsst.multi INFO: found job 16971861 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:09 proc_lsst.multi:146 [INFO] found job 16971862 in provider astro proc_lsst.multi INFO: found job 16971862 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:09 proc_lsst.multi:146 [INFO] found job 16971863 in provider astro proc_lsst.multi INFO: found job 16971863 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:10 proc_lsst.multi:146 [INFO] found job 16971864 in provider astro proc_lsst.multi INFO: found job 16971864 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.dataflow.dflow INFO: Task 147 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4426568f1d102c4fa6d9330c3acc7285 with result from task 147 parsl.dataflow.dflow DEBUG: Task 147 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/9c0c085e-dcc4-4e1b-820c-dce455d7ae8e_isrForCrosstalkSources_860802_2.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/9c0c085e-dcc4-4e1b-820c-dce455d7ae8e_isrForCrosstalkSources_860802_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:10 proc_lsst.multi:146 [INFO] found job 16971865 in provider astro proc_lsst.multi INFO: found job 16971865 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:11 proc_lsst.multi:146 [INFO] found job 16971866 in provider astro proc_lsst.multi INFO: found job 16971866 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.dataflow.dflow INFO: Task 145 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 723e56fbcb99b092576eb8a5b88a68f9 with result from task 145 parsl.dataflow.dflow DEBUG: Task 145 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/c74c5d3c-a990-4a76-9e2c-b7a3d8d957a8_isrForCrosstalkSources_860803_51.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/c74c5d3c-a990-4a76-9e2c-b7a3d8d957a8_isrForCrosstalkSources_860803_51.stderr parsl.dataflow.dflow INFO: Task 144 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79f85800406f6ce1d6e0c4f9789d2d0e with result from task 144 parsl.dataflow.dflow DEBUG: Task 144 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/00c5577e-0078-4f29-8a5f-6777901d9c3d_isrForCrosstalkSources_860802_48.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/00c5577e-0078-4f29-8a5f-6777901d9c3d_isrForCrosstalkSources_860802_48.stderr parsl.dataflow.dflow INFO: Task 143 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6fbb106b19e1fed70d377b6600acafb with result from task 143 parsl.dataflow.dflow DEBUG: Task 143 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/e89327c2-d9f7-4a4b-b26f-3fb2ea0b08ef_isrForCrosstalkSources_860802_57.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/e89327c2-d9f7-4a4b-b26f-3fb2ea0b08ef_isrForCrosstalkSources_860802_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:11 proc_lsst.multi:146 [INFO] found job 16971867 in provider astro proc_lsst.multi INFO: found job 16971867 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.dataflow.dflow INFO: Task 146 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e8fe896d2d8d533124bf128b4213504 with result from task 146 parsl.dataflow.dflow DEBUG: Task 146 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/7f3f7278-121d-41e5-881a-038541fec844_isrForCrosstalkSources_860802_41.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/7f3f7278-121d-41e5-881a-038541fec844_isrForCrosstalkSources_860802_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:11 proc_lsst.multi:146 [INFO] found job 16971868 in provider astro proc_lsst.multi INFO: found job 16971868 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:12 proc_lsst.multi:146 [INFO] found job 16971869 in provider astro proc_lsst.multi INFO: found job 16971869 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:12 proc_lsst.multi:146 [INFO] found job 16971870 in provider astro proc_lsst.multi INFO: found job 16971870 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:12 proc_lsst.multi:146 [INFO] found job 16971871 in provider astro proc_lsst.multi INFO: found job 16971871 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:12 proc_lsst.multi:146 [INFO] found job 16971872 in provider escience_rtx6k proc_lsst.multi INFO: found job 16971872 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971872,16971873,16971874,16971875,16971877' parsl.providers.slurm.slurm DEBUG: squeue returned 16971877 R 16971872 R 16971873 R 16971874 R 16971875 R parsl.providers.slurm.slurm DEBUG: Updating job 16971877 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971872 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971873 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971874 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971875 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:13 proc_lsst.multi:146 [INFO] found job 16971873 in provider escience_rtx6k proc_lsst.multi INFO: found job 16971873 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971872,16971873,16971874,16971875,16971877' parsl.dataflow.dflow INFO: Task 156 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2740280555ece9376a046b56f106e3fa with result from task 156 parsl.dataflow.dflow DEBUG: Task 156 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/50d61ceb-9e4a-47db-ad01-d00a6f2288b2_isrForCrosstalkSources_860802_45.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/50d61ceb-9e4a-47db-ad01-d00a6f2288b2_isrForCrosstalkSources_860802_45.stderr parsl.dataflow.dflow INFO: Task 155 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f5ada2967edee107ebf05db61be56b0 with result from task 155 parsl.dataflow.dflow DEBUG: Task 155 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/6f51b79f-6c0b-46c2-a3de-f58bb74bef6a_isrForCrosstalkSources_860801_33.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/6f51b79f-6c0b-46c2-a3de-f58bb74bef6a_isrForCrosstalkSources_860801_33.stderr parsl.dataflow.dflow INFO: Task 151 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3fac9f03fd0d0bcfb16c9e7e5b23b588 with result from task 151 parsl.dataflow.dflow DEBUG: Task 151 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/c6427985-d3f6-4646-bdf8-fe311d92eb29_isrForCrosstalkSources_860802_31.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/c6427985-d3f6-4646-bdf8-fe311d92eb29_isrForCrosstalkSources_860802_31.stderr parsl.dataflow.dflow INFO: Task 154 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c898aef8f19809b04e7953e32020f143 with result from task 154 parsl.dataflow.dflow DEBUG: Task 154 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/18b52ba3-4ed7-435d-87d3-3367e2d098a8_isrForCrosstalkSources_860802_34.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/18b52ba3-4ed7-435d-87d3-3367e2d098a8_isrForCrosstalkSources_860802_34.stderr parsl.dataflow.dflow INFO: Task 157 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1f8ebbd5b9e9f0f19fb8ff1b94f9f4a with result from task 157 parsl.dataflow.dflow DEBUG: Task 157 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/0172fc64-d898-46c9-bbd0-32c59b7d415a_isrForCrosstalkSources_860802_52.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/0172fc64-d898-46c9-bbd0-32c59b7d415a_isrForCrosstalkSources_860802_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971877 R 16971872 R 16971873 R 16971874 R 16971875 R parsl.providers.slurm.slurm DEBUG: Updating job 16971877 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971872 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971873 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971874 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971875 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:13 proc_lsst.multi:146 [INFO] found job 16971874 in provider escience_rtx6k proc_lsst.multi INFO: found job 16971874 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971872,16971873,16971874,16971875,16971877' parsl.dataflow.dflow INFO: Task 158 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a19837748da61df3a481a786580de24e with result from task 158 parsl.dataflow.dflow DEBUG: Task 158 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/a7f3ca5b-e36d-4b7d-9525-bc1057269598_isrForCrosstalkSources_860801_36.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/a7f3ca5b-e36d-4b7d-9525-bc1057269598_isrForCrosstalkSources_860801_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971877 R 16971872 R 16971873 R 16971874 R 16971875 R parsl.providers.slurm.slurm DEBUG: Updating job 16971877 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971872 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971873 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971874 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971875 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:13 proc_lsst.multi:146 [INFO] found job 16971875 in provider escience_rtx6k proc_lsst.multi INFO: found job 16971875 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971872,16971873,16971874,16971875,16971877' parsl.dataflow.dflow INFO: Task 152 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ccaaf4a52a77da76ec9867c4142d281 with result from task 152 parsl.dataflow.dflow DEBUG: Task 152 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/6e0817a2-fc92-4f03-9f83-f1794a05a75e_isrForCrosstalkSources_860801_20.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/6e0817a2-fc92-4f03-9f83-f1794a05a75e_isrForCrosstalkSources_860801_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971877 R 16971872 R 16971873 R 16971874 R 16971875 R parsl.providers.slurm.slurm DEBUG: Updating job 16971877 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971872 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971873 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971874 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971875 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:13 proc_lsst.multi:146 [INFO] found job 16971877 in provider escience_rtx6k proc_lsst.multi INFO: found job 16971877 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971872,16971873,16971874,16971875,16971877' parsl.providers.slurm.slurm DEBUG: squeue returned 16971877 R 16971872 R 16971873 R 16971874 R 16971875 R parsl.providers.slurm.slurm DEBUG: Updating job 16971877 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971872 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971873 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971874 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971875 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:14 proc_lsst.multi:146 [INFO] found job 16971878 in provider escience_a40 proc_lsst.multi INFO: found job 16971878 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.dataflow.dflow INFO: Task 159 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f00e532dd159e4489e66158cb1658239 with result from task 159 parsl.dataflow.dflow DEBUG: Task 159 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/3e0cfb2c-a231-4a12-a2dc-81cb26f7970e_isrForCrosstalkSources_860801_16.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/3e0cfb2c-a231-4a12-a2dc-81cb26f7970e_isrForCrosstalkSources_860801_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:14 proc_lsst.multi:146 [INFO] found job 16971879 in provider escience_a40 proc_lsst.multi INFO: found job 16971879 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:14 proc_lsst.multi:146 [INFO] found job 16971880 in provider escience_a40 proc_lsst.multi INFO: found job 16971880 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.dataflow.dflow INFO: Task 149 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a1ce2fea28dbf63ad913ad4f5f7afd0 with result from task 149 parsl.dataflow.dflow DEBUG: Task 149 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/b9eb3d40-df89-4329-89c3-fa1bd6dc1419_isrForCrosstalkSources_860803_19.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/b9eb3d40-df89-4329-89c3-fa1bd6dc1419_isrForCrosstalkSources_860803_19.stderr parsl.dataflow.dflow INFO: Task 148 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb5901ed24b0ba17c180dcff23b30c41 with result from task 148 parsl.dataflow.dflow DEBUG: Task 148 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/0898a38a-def6-476a-ad7e-828b92edaa39_isrForCrosstalkSources_860801_30.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/0898a38a-def6-476a-ad7e-828b92edaa39_isrForCrosstalkSources_860801_30.stderr parsl.dataflow.dflow INFO: Task 161 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f998a9515d646a0fb561980668e9c3a with result from task 161 parsl.dataflow.dflow DEBUG: Task 161 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/beac8950-e65f-49ac-9036-833a11e1722b_isrForCrosstalkSources_860802_32.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/beac8950-e65f-49ac-9036-833a11e1722b_isrForCrosstalkSources_860802_32.stderr parsl.dataflow.dflow INFO: Task 162 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f698cbc222a732e195557216910bd72d with result from task 162 parsl.dataflow.dflow DEBUG: Task 162 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/205957c0-a6dc-4b69-b788-51bce0808c6a_isrForCrosstalkSources_860802_23.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/205957c0-a6dc-4b69-b788-51bce0808c6a_isrForCrosstalkSources_860802_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:15 proc_lsst.multi:146 [INFO] found job 16971881 in provider escience_a40 proc_lsst.multi INFO: found job 16971881 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:15 proc_lsst.multi:146 [INFO] found job 16971882 in provider escience_a40 proc_lsst.multi INFO: found job 16971882 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.dataflow.dflow INFO: Task 106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8989a44e0d7225152069b2499256fef8 with result from task 106 parsl.dataflow.dflow DEBUG: Task 106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/2f1c2e08-bb85-4b39-a3e3-63f8bb40ef54_isrForCrosstalkSources_860803_17.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/2f1c2e08-bb85-4b39-a3e3-63f8bb40ef54_isrForCrosstalkSources_860803_17.stderr parsl.dataflow.dflow INFO: Task 163 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ef1dfdf0d199a093b594d6d5b29b005 with result from task 163 parsl.dataflow.dflow DEBUG: Task 163 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/61110114-d436-48ea-b9bc-97c5c25c5a53_isrForCrosstalkSources_860802_13.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/61110114-d436-48ea-b9bc-97c5c25c5a53_isrForCrosstalkSources_860802_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:15 proc_lsst.multi:146 [INFO] found job 16971883 in provider escience_a40 proc_lsst.multi INFO: found job 16971883 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.dataflow.dflow INFO: Task 165 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef12ecbe7860d24b73cd3a3ba4d17b03 with result from task 165 parsl.dataflow.dflow DEBUG: Task 165 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/cf9777a9-a65f-4bff-a8f1-e87f96dc1796_isrForCrosstalkSources_860801_61.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/cf9777a9-a65f-4bff-a8f1-e87f96dc1796_isrForCrosstalkSources_860801_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:15 proc_lsst.multi:146 [INFO] found job 16971884 in provider escience_a40 proc_lsst.multi INFO: found job 16971884 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:16 proc_lsst.multi:146 [INFO] found job 16971885 in provider escience_a40 proc_lsst.multi INFO: found job 16971885 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:16 proc_lsst.multi:146 [INFO] found job 16971886 in provider escience_a40 proc_lsst.multi INFO: found job 16971886 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.dataflow.dflow INFO: Task 150 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 121366f266bc31c1f1418033d5f12dd5 with result from task 150 parsl.dataflow.dflow DEBUG: Task 150 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/7d5ebc1c-c542-403c-9344-7d0356b96790_isrForCrosstalkSources_860803_39.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/7d5ebc1c-c542-403c-9344-7d0356b96790_isrForCrosstalkSources_860803_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:16 proc_lsst.multi:146 [INFO] found job 16971887 in provider escience_a40 proc_lsst.multi INFO: found job 16971887 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:16 proc_lsst.multi:146 [INFO] found job 16971888 in provider escience_a40 proc_lsst.multi INFO: found job 16971888 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 27 parsl.dataflow.strategy DEBUG: Executor multi has 27 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23162860286032 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 27 parsl.dataflow.strategy DEBUG: Executor multi has 27 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23162860286032 parsl.dataflow.dflow INFO: Task 185 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c03ffb20fb27c157f278fc82ee143616 with result from task 185 parsl.dataflow.dflow DEBUG: Task 185 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/86411d0f-3d53-41a2-be13-e0a6e503dcc7_isrForCrosstalkSources_860803_31.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/86411d0f-3d53-41a2-be13-e0a6e503dcc7_isrForCrosstalkSources_860803_31.stderr parsl.dataflow.dflow INFO: Task 153 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1cff7dcda069cb91130bd4a66b8fe949 with result from task 153 parsl.dataflow.dflow DEBUG: Task 153 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/95cc8e5e-97dc-4d0b-b91f-16fffa5f1116_isrForCrosstalkSources_860802_27.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/95cc8e5e-97dc-4d0b-b91f-16fffa5f1116_isrForCrosstalkSources_860802_27.stderr parsl.dataflow.dflow INFO: Task 160 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba907e883caf7c6152232aaba0cf6585 with result from task 160 parsl.dataflow.dflow DEBUG: Task 160 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/399547a3-5e9b-45f5-ab86-24246476e7f9_isrForCrosstalkSources_860801_47.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/399547a3-5e9b-45f5-ab86-24246476e7f9_isrForCrosstalkSources_860801_47.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 24 parsl.dataflow.strategy DEBUG: Executor multi has 24 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23162860286032 2024-03-10 01:40:27 proc_lsst.multi:146 [INFO] found job 44401 in provider local proc_lsst.multi INFO: found job 44401 in provider local 2024-03-10 01:40:27 proc_lsst.multi:146 [INFO] found job 16971842 in provider astro proc_lsst.multi INFO: found job 16971842 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:27 proc_lsst.multi:146 [INFO] found job 16971843 in provider astro proc_lsst.multi INFO: found job 16971843 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:27 proc_lsst.multi:146 [INFO] found job 16971844 in provider astro proc_lsst.multi INFO: found job 16971844 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:27 proc_lsst.multi:146 [INFO] found job 16971845 in provider astro proc_lsst.multi INFO: found job 16971845 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:28 proc_lsst.multi:146 [INFO] found job 16971846 in provider astro proc_lsst.multi INFO: found job 16971846 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:28 proc_lsst.multi:146 [INFO] found job 16971847 in provider astro proc_lsst.multi INFO: found job 16971847 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:28 proc_lsst.multi:146 [INFO] found job 16971848 in provider astro proc_lsst.multi INFO: found job 16971848 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:28 proc_lsst.multi:146 [INFO] found job 16971849 in provider astro proc_lsst.multi INFO: found job 16971849 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:28 proc_lsst.multi:146 [INFO] found job 16971850 in provider astro proc_lsst.multi INFO: found job 16971850 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.dataflow.dflow INFO: Task 94 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57d5d8866391f8b76102844f48902a0b with result from task 94 parsl.dataflow.dflow DEBUG: Task 94 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/27a38fe1-9277-40f1-9596-30050bfe6161_isrForCrosstalkSources_860803_7.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/27a38fe1-9277-40f1-9596-30050bfe6161_isrForCrosstalkSources_860803_7.stderr parsl.dataflow.dflow INFO: Task 97 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75101a036655cf881fd268b05034fdae with result from task 97 parsl.dataflow.dflow DEBUG: Task 97 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/5bcbe701-8194-45ed-b3eb-c09fba563b78_isrForCrosstalkSources_860802_19.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/5bcbe701-8194-45ed-b3eb-c09fba563b78_isrForCrosstalkSources_860802_19.stderr parsl.dataflow.dflow INFO: Task 96 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a92012a12442f7c5056cdbaeddc07a1 with result from task 96 parsl.dataflow.dflow DEBUG: Task 96 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/a82613c5-0104-4dd4-91a2-b6f6551bd42e_isrForCrosstalkSources_860802_9.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/a82613c5-0104-4dd4-91a2-b6f6551bd42e_isrForCrosstalkSources_860802_9.stderr parsl.dataflow.dflow INFO: Task 95 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4892b210c41fb6a9dba054845eed5193 with result from task 95 parsl.dataflow.dflow DEBUG: Task 95 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/f7694f75-90c6-4b88-82ad-17968c4c847a_isrForCrosstalkSources_860803_21.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/f7694f75-90c6-4b88-82ad-17968c4c847a_isrForCrosstalkSources_860803_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:29 proc_lsst.multi:146 [INFO] found job 16971851 in provider astro proc_lsst.multi INFO: found job 16971851 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:29 proc_lsst.multi:146 [INFO] found job 16971852 in provider astro proc_lsst.multi INFO: found job 16971852 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:29 proc_lsst.multi:146 [INFO] found job 16971853 in provider astro proc_lsst.multi INFO: found job 16971853 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:29 proc_lsst.multi:146 [INFO] found job 16971854 in provider astro proc_lsst.multi INFO: found job 16971854 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:29 proc_lsst.multi:146 [INFO] found job 16971855 in provider astro proc_lsst.multi INFO: found job 16971855 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:30 proc_lsst.multi:146 [INFO] found job 16971856 in provider astro proc_lsst.multi INFO: found job 16971856 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:30 proc_lsst.multi:146 [INFO] found job 16971857 in provider astro proc_lsst.multi INFO: found job 16971857 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:30 proc_lsst.multi:146 [INFO] found job 16971858 in provider astro proc_lsst.multi INFO: found job 16971858 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:30 proc_lsst.multi:146 [INFO] found job 16971859 in provider astro proc_lsst.multi INFO: found job 16971859 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:31 proc_lsst.multi:146 [INFO] found job 16971860 in provider astro proc_lsst.multi INFO: found job 16971860 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:31 proc_lsst.multi:146 [INFO] found job 16971861 in provider astro proc_lsst.multi INFO: found job 16971861 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:31 proc_lsst.multi:146 [INFO] found job 16971862 in provider astro proc_lsst.multi INFO: found job 16971862 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:31 proc_lsst.multi:146 [INFO] found job 16971863 in provider astro proc_lsst.multi INFO: found job 16971863 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:31 proc_lsst.multi:146 [INFO] found job 16971864 in provider astro proc_lsst.multi INFO: found job 16971864 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:32 proc_lsst.multi:146 [INFO] found job 16971865 in provider astro proc_lsst.multi INFO: found job 16971865 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:32 proc_lsst.multi:146 [INFO] found job 16971866 in provider astro proc_lsst.multi INFO: found job 16971866 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:32 proc_lsst.multi:146 [INFO] found job 16971867 in provider astro proc_lsst.multi INFO: found job 16971867 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:32 proc_lsst.multi:146 [INFO] found job 16971868 in provider astro proc_lsst.multi INFO: found job 16971868 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:33 proc_lsst.multi:146 [INFO] found job 16971869 in provider astro proc_lsst.multi INFO: found job 16971869 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:33 proc_lsst.multi:146 [INFO] found job 16971870 in provider astro proc_lsst.multi INFO: found job 16971870 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:33 proc_lsst.multi:146 [INFO] found job 16971871 in provider astro proc_lsst.multi INFO: found job 16971871 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971842,16971843,16971844,16971845,16971846,16971847,16971848,16971849,16971850,16971851,16971852,16971853,16971854,16971855,16971856,16971857,16971858,16971859,16971860,16971861,16971862,16971863,16971864,16971865,16971866,16971867,16971868,16971869,16971870,16971871' parsl.providers.slurm.slurm DEBUG: squeue returned 16971868 R 16971869 R 16971870 R 16971871 R 16971848 R 16971849 R 16971850 R 16971851 R 16971852 R 16971853 R 16971854 R 16971855 R 16971856 R 16971857 R 16971858 R 16971859 R 16971860 R 16971861 R 16971862 R 16971863 R 16971864 R 16971865 R 16971866 R 16971867 R 16971844 R 16971845 R 16971846 R 16971847 R 16971842 R 16971843 R parsl.providers.slurm.slurm DEBUG: Updating job 16971868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971849 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971850 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971851 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971857 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971861 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971843 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:33 proc_lsst.multi:146 [INFO] found job 16971872 in provider escience_rtx6k proc_lsst.multi INFO: found job 16971872 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971872,16971873,16971874,16971875,16971877' parsl.providers.slurm.slurm DEBUG: squeue returned 16971877 R 16971872 R 16971873 R 16971874 R 16971875 R parsl.providers.slurm.slurm DEBUG: Updating job 16971877 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971872 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971873 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971874 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971875 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:33 proc_lsst.multi:146 [INFO] found job 16971873 in provider escience_rtx6k proc_lsst.multi INFO: found job 16971873 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971872,16971873,16971874,16971875,16971877' parsl.providers.slurm.slurm DEBUG: squeue returned 16971877 R 16971872 R 16971873 R 16971874 R 16971875 R parsl.providers.slurm.slurm DEBUG: Updating job 16971877 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971872 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971873 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971874 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971875 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:33 proc_lsst.multi:146 [INFO] found job 16971874 in provider escience_rtx6k proc_lsst.multi INFO: found job 16971874 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971872,16971873,16971874,16971875,16971877' parsl.providers.slurm.slurm DEBUG: squeue returned 16971877 R 16971872 R 16971873 R 16971874 R 16971875 R parsl.providers.slurm.slurm DEBUG: Updating job 16971877 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971872 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971873 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971874 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971875 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:36 proc_lsst.multi:146 [INFO] found job 16971875 in provider escience_rtx6k proc_lsst.multi INFO: found job 16971875 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971872,16971873,16971874,16971875,16971877' parsl.providers.slurm.slurm DEBUG: squeue returned 16971877 R 16971872 R 16971873 R 16971874 R 16971875 R parsl.providers.slurm.slurm DEBUG: Updating job 16971877 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971872 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971873 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971874 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971875 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:36 proc_lsst.multi:146 [INFO] found job 16971877 in provider escience_rtx6k proc_lsst.multi INFO: found job 16971877 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971872,16971873,16971874,16971875,16971877' parsl.providers.slurm.slurm DEBUG: squeue returned 16971877 R 16971872 R 16971873 R 16971874 R 16971875 R parsl.providers.slurm.slurm DEBUG: Updating job 16971877 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971872 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971873 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971874 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971875 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:36 proc_lsst.multi:146 [INFO] found job 16971878 in provider escience_a40 proc_lsst.multi INFO: found job 16971878 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:36 proc_lsst.multi:146 [INFO] found job 16971879 in provider escience_a40 proc_lsst.multi INFO: found job 16971879 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:36 proc_lsst.multi:146 [INFO] found job 16971880 in provider escience_a40 proc_lsst.multi INFO: found job 16971880 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:36 proc_lsst.multi:146 [INFO] found job 16971881 in provider escience_a40 proc_lsst.multi INFO: found job 16971881 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:37 proc_lsst.multi:146 [INFO] found job 16971882 in provider escience_a40 proc_lsst.multi INFO: found job 16971882 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:37 proc_lsst.multi:146 [INFO] found job 16971883 in provider escience_a40 proc_lsst.multi INFO: found job 16971883 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:37 proc_lsst.multi:146 [INFO] found job 16971884 in provider escience_a40 proc_lsst.multi INFO: found job 16971884 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:37 proc_lsst.multi:146 [INFO] found job 16971885 in provider escience_a40 proc_lsst.multi INFO: found job 16971885 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:37 proc_lsst.multi:146 [INFO] found job 16971886 in provider escience_a40 proc_lsst.multi INFO: found job 16971886 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:38 proc_lsst.multi:146 [INFO] found job 16971887 in provider escience_a40 proc_lsst.multi INFO: found job 16971887 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING 2024-03-10 01:40:38 proc_lsst.multi:146 [INFO] found job 16971888 in provider escience_a40 proc_lsst.multi INFO: found job 16971888 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16971878,16971879,16971880,16971881,16971882,16971883,16971884,16971885,16971886,16971887,16971888' parsl.providers.slurm.slurm DEBUG: squeue returned 16971888 R 16971883 R 16971884 R 16971885 R 16971886 R 16971887 R 16971878 R 16971879 R 16971880 R 16971881 R 16971882 R parsl.providers.slurm.slurm DEBUG: Updating job 16971888 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971883 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971884 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971885 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971886 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971887 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971879 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971881 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16971882 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 20 parsl.dataflow.strategy DEBUG: Executor multi has 20 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23162860286032 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 20 parsl.dataflow.strategy DEBUG: Executor multi has 20 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23162860286032 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 20 parsl.dataflow.strategy DEBUG: Executor multi has 20 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23162860286032 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 20 parsl.dataflow.strategy DEBUG: Executor multi has 20 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23162860286032 parsl.dataflow.dflow INFO: Task 178 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4d9a7449d7da6dca8f3fbfc17b3c5db with result from task 178 parsl.dataflow.dflow DEBUG: Task 178 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/3857ff2a-600b-4a75-8908-58f6f129342f_isrForCrosstalkSources_860803_26.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/3857ff2a-600b-4a75-8908-58f6f129342f_isrForCrosstalkSources_860803_26.stderr parsl.dataflow.dflow INFO: Task 175 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 942875531bf4b6f47ae518556769fc36 with result from task 175 parsl.dataflow.dflow DEBUG: Task 175 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/c3736c01-59ac-4475-9770-f4239d498e04_isrForCrosstalkSources_860801_21.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/c3736c01-59ac-4475-9770-f4239d498e04_isrForCrosstalkSources_860801_21.stderr parsl.dataflow.dflow INFO: Task 183 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c4a16e56d9898c64278b0597dc4863c with result from task 183 parsl.dataflow.dflow DEBUG: Task 183 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/9176a5d3-372d-4f4e-9f34-ead222248b62_isrForCrosstalkSources_860802_46.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/9176a5d3-372d-4f4e-9f34-ead222248b62_isrForCrosstalkSources_860802_46.stderr parsl.dataflow.dflow INFO: Task 172 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af0bc02e0f12d1cc99788f413908d8f3 with result from task 172 parsl.dataflow.dflow DEBUG: Task 172 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/49260c23-be57-46fa-aa07-eddbd804d1f1_isrForCrosstalkSources_860802_40.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/49260c23-be57-46fa-aa07-eddbd804d1f1_isrForCrosstalkSources_860802_40.stderr parsl.dataflow.dflow INFO: Task 174 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1b276787a563f92e30e3d4e45f296b7 with result from task 174 parsl.dataflow.dflow DEBUG: Task 174 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/54203813-1fd9-4a12-86c1-e5a583dd1337_isrForCrosstalkSources_860801_39.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/54203813-1fd9-4a12-86c1-e5a583dd1337_isrForCrosstalkSources_860801_39.stderr parsl.dataflow.dflow INFO: Task 169 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba95052539a23860b978056a96d2d432 with result from task 169 parsl.dataflow.dflow DEBUG: Task 169 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/1cecc060-209e-4469-8abd-2c5a88865479_isrForCrosstalkSources_860802_20.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/1cecc060-209e-4469-8abd-2c5a88865479_isrForCrosstalkSources_860802_20.stderr parsl.dataflow.dflow INFO: Task 184 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d480f47966a2111277df580619e5070 with result from task 184 parsl.dataflow.dflow DEBUG: Task 184 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/fc96a945-5135-42b3-8f24-3a0ae62d6e21_isrForCrosstalkSources_860801_51.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/fc96a945-5135-42b3-8f24-3a0ae62d6e21_isrForCrosstalkSources_860801_51.stderr parsl.dataflow.dflow INFO: Task 167 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0413362d7e294575446d83cf24977417 with result from task 167 parsl.dataflow.dflow DEBUG: Task 167 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/50950038-3566-4675-8af9-c8a45660883b_isrForCrosstalkSources_860802_15.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/50950038-3566-4675-8af9-c8a45660883b_isrForCrosstalkSources_860802_15.stderr parsl.dataflow.dflow INFO: Task 177 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3aedb4a233a80f1ae7b3c14aebaa89bc with result from task 177 parsl.dataflow.dflow DEBUG: Task 177 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/19bb75c2-8ac0-42c3-ada0-d8901d8715c4_isrForCrosstalkSources_860802_16.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/19bb75c2-8ac0-42c3-ada0-d8901d8715c4_isrForCrosstalkSources_860802_16.stderr parsl.dataflow.dflow INFO: Task 171 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6139d722d6bb6314f9b78d1776a6151 with result from task 171 parsl.dataflow.dflow DEBUG: Task 171 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/274e5d19-82cb-4829-b992-9c16857efe7a_isrForCrosstalkSources_860801_15.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/274e5d19-82cb-4829-b992-9c16857efe7a_isrForCrosstalkSources_860801_15.stderr parsl.dataflow.dflow INFO: Task 166 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 042d1a61aae358d291aba098cfb622b0 with result from task 166 parsl.dataflow.dflow DEBUG: Task 166 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/543ed78e-36ea-495f-98d0-019f3b6df0b6_isrForCrosstalkSources_860803_11.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/543ed78e-36ea-495f-98d0-019f3b6df0b6_isrForCrosstalkSources_860803_11.stderr parsl.dataflow.dflow INFO: Task 168 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 199df6efa6f80155d45dfc600f505a4d with result from task 168 parsl.dataflow.dflow DEBUG: Task 168 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/74253bf0-ffc2-44f1-806d-3fb0e3332a21_isrForCrosstalkSources_860801_12.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/74253bf0-ffc2-44f1-806d-3fb0e3332a21_isrForCrosstalkSources_860801_12.stderr parsl.dataflow.dflow INFO: Task 176 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 877440132e04c21ef4981a252f9c0055 with result from task 176 parsl.dataflow.dflow DEBUG: Task 176 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/3a93d494-8a79-4ea0-97b4-2551c7bf94d2_isrForCrosstalkSources_860802_6.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860802/3a93d494-8a79-4ea0-97b4-2551c7bf94d2_isrForCrosstalkSources_860802_6.stderr parsl.dataflow.dflow INFO: Task 181 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8f56d1271afe455c8220c2a7f61246b with result from task 181 parsl.dataflow.dflow DEBUG: Task 181 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/a6062270-47cd-4cdf-a3dd-607309d5d4d0_isrForCrosstalkSources_860801_37.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/a6062270-47cd-4cdf-a3dd-607309d5d4d0_isrForCrosstalkSources_860801_37.stderr parsl.dataflow.dflow INFO: Task 182 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e47f1ea806d8e4c67db22b76e29c323 with result from task 182 parsl.dataflow.dflow DEBUG: Task 182 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/72711c40-2aab-4425-a368-9e30ed09c2ae_isrForCrosstalkSources_860803_50.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/72711c40-2aab-4425-a368-9e30ed09c2ae_isrForCrosstalkSources_860803_50.stderr parsl.dataflow.dflow INFO: Task 179 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2af0f58df387d85a9d1782ff00919732 with result from task 179 parsl.dataflow.dflow DEBUG: Task 179 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/f945a938-0032-4edd-a0cf-50e35a92acc1_isrForCrosstalkSources_860803_36.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/f945a938-0032-4edd-a0cf-50e35a92acc1_isrForCrosstalkSources_860803_36.stderr parsl.dataflow.dflow INFO: Task 164 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d4ff401e15d6e2cc1195b1dffa33895 with result from task 164 parsl.dataflow.dflow DEBUG: Task 164 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/84aa977f-c390-49bc-9c63-9737468315be_isrForCrosstalkSources_860803_49.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/84aa977f-c390-49bc-9c63-9737468315be_isrForCrosstalkSources_860803_49.stderr parsl.dataflow.dflow INFO: Task 170 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7458c977ee91028adea10bc5bb2778f with result from task 170 parsl.dataflow.dflow DEBUG: Task 170 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/55d81396-0c3c-41b6-a16f-87f84ce4e007_isrForCrosstalkSources_860803_38.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/55d81396-0c3c-41b6-a16f-87f84ce4e007_isrForCrosstalkSources_860803_38.stderr parsl.dataflow.dflow INFO: Task 173 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38a0a75566e6d42170c61dd3413e6c40 with result from task 173 parsl.dataflow.dflow DEBUG: Task 173 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/74bcbe5d-da66-4521-b66a-1abc02317ab5_isrForCrosstalkSources_860801_57.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860801/74bcbe5d-da66-4521-b66a-1abc02317ab5_isrForCrosstalkSources_860801_57.stderr parsl.dataflow.dflow INFO: Task 180 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c1a55ab9784d0e66cd1ec684e6a0077 with result from task 180 parsl.dataflow.dflow INFO: DFK cleanup initiated parsl.dataflow.dflow INFO: Summary of tasks in DFK: parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0 parsl.dataflow.dflow INFO: Tasks in state States.pending: 0 parsl.dataflow.dflow DEBUG: Task 180 checkpointed parsl.dataflow.dflow INFO: Tasks in state States.running: 0 parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 186 parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Tasks in state States.failed: 0 parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/fcc57517-6c72-4b38-85cc-1a0f544e47ea_isrForCrosstalkSources_860803_61.stdout parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 0 parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/submit/DEEP/20190601/A1c/science#step0/20240310T013701Z/logs/isrForCrosstalkSources/860803/fcc57517-6c72-4b38-85cc-1a0f544e47ea_isrForCrosstalkSources_860803_61.stderr parsl.dataflow.dflow INFO: Tasks in state States.launched: 0 parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0 parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0 parsl.dataflow.dflow INFO: Tasks in state States.joining: 0 parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0 parsl.dataflow.dflow INFO: End of summary parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Closing job status poller parsl.dataflow.dflow INFO: Terminated job status poller parsl.dataflow.dflow INFO: Scaling in and shutting down executors parsl.dataflow.dflow INFO: Scaling in executor multi parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=47, block_ids=[] parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 47 blocks parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f17763866586 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f17763866586 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cb66e641a742 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cb66e641a742 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7f8a9fe3d696 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7f8a9fe3d696 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bf57548c3d4b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bf57548c3d4b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 856f339fef5e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 856f339fef5e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8a6b488ca14b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8a6b488ca14b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 71a510c4cfc6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 71a510c4cfc6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 98a86dd1b8b3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 98a86dd1b8b3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 54d98907fc22 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 54d98907fc22 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4d156c6931a6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4d156c6931a6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 14cd42db5c85 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 14cd42db5c85 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 242ab745690c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 242ab745690c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e01f4d8d69fa parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e01f4d8d69fa parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ad82cbea1ed4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ad82cbea1ed4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 05dfab6fd75f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 05dfab6fd75f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ba848692982a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ba848692982a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 38217e63471b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 38217e63471b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4533547e9e21 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4533547e9e21 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b47748d72694 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b47748d72694 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6e7bac56ee1f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6e7bac56ee1f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1201836acf6f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1201836acf6f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: acd1102b4c56 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: acd1102b4c56 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dd44296d0d7a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dd44296d0d7a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c77ca63ec53c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c77ca63ec53c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ce8d50b0bbb4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ce8d50b0bbb4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 504a1a4186da parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 504a1a4186da parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4c2193418d2c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4c2193418d2c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 262598bf922b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 262598bf922b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: db6af84bca41 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: db6af84bca41 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ae1b752748e8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ae1b752748e8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 15c5ba40b182 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 15c5ba40b182 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 89266a7865a8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 89266a7865a8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 504e8c628287 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 504e8c628287 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 558ad4bd0ccd parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 558ad4bd0ccd parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cd853e98c027 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cd853e98c027 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6aea000b500a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6aea000b500a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1a7c5b5f6a41 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1a7c5b5f6a41 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 620adbea1bb8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 620adbea1bb8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 137a25b2d0f5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 137a25b2d0f5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8d870c885aff parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8d870c885aff parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3e8c263e2b97 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3e8c263e2b97 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8f1c245c7a89 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8f1c245c7a89 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6b57537540a2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6b57537540a2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6d6b23479e1a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6d6b23479e1a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bef8aabdb386 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bef8aabdb386 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 04c33f79223c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 04c33f79223c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 61c7b518c327 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 61c7b518c327 2024-03-10 01:40:52 proc_lsst.multi:146 [INFO] found job 16971861 in provider astro proc_lsst.multi INFO: found job 16971861 in provider astro 2024-03-10 01:40:52 proc_lsst.multi:201 [INFO] cancelling 16971861 on provider astro proc_lsst.multi INFO: cancelling 16971861 on provider astro 2024-03-10 01:40:52 proc_lsst.multi:146 [INFO] found job 16971858 in provider astro proc_lsst.multi INFO: found job 16971858 in provider astro 2024-03-10 01:40:52 proc_lsst.multi:201 [INFO] cancelling 16971858 on provider astro proc_lsst.multi INFO: cancelling 16971858 on provider astro 2024-03-10 01:40:52 proc_lsst.multi:146 [INFO] found job 16971849 in provider astro proc_lsst.multi INFO: found job 16971849 in provider astro 2024-03-10 01:40:52 proc_lsst.multi:201 [INFO] cancelling 16971849 on provider astro proc_lsst.multi INFO: cancelling 16971849 on provider astro 2024-03-10 01:40:52 proc_lsst.multi:146 [INFO] found job 16971859 in provider astro proc_lsst.multi INFO: found job 16971859 in provider astro 2024-03-10 01:40:52 proc_lsst.multi:201 [INFO] cancelling 16971859 on provider astro proc_lsst.multi INFO: cancelling 16971859 on provider astro 2024-03-10 01:40:52 proc_lsst.multi:146 [INFO] found job 16971864 in provider astro proc_lsst.multi INFO: found job 16971864 in provider astro 2024-03-10 01:40:52 proc_lsst.multi:201 [INFO] cancelling 16971864 on provider astro proc_lsst.multi INFO: cancelling 16971864 on provider astro 2024-03-10 01:40:53 proc_lsst.multi:146 [INFO] found job 16971856 in provider astro proc_lsst.multi INFO: found job 16971856 in provider astro 2024-03-10 01:40:53 proc_lsst.multi:201 [INFO] cancelling 16971856 on provider astro proc_lsst.multi INFO: cancelling 16971856 on provider astro 2024-03-10 01:40:53 proc_lsst.multi:146 [INFO] found job 16971854 in provider astro proc_lsst.multi INFO: found job 16971854 in provider astro 2024-03-10 01:40:53 proc_lsst.multi:201 [INFO] cancelling 16971854 on provider astro proc_lsst.multi INFO: cancelling 16971854 on provider astro 2024-03-10 01:40:53 proc_lsst.multi:146 [INFO] found job 16971850 in provider astro proc_lsst.multi INFO: found job 16971850 in provider astro 2024-03-10 01:40:53 proc_lsst.multi:201 [INFO] cancelling 16971850 on provider astro proc_lsst.multi INFO: cancelling 16971850 on provider astro 2024-03-10 01:40:53 proc_lsst.multi:146 [INFO] found job 16971866 in provider astro proc_lsst.multi INFO: found job 16971866 in provider astro 2024-03-10 01:40:53 proc_lsst.multi:201 [INFO] cancelling 16971866 on provider astro proc_lsst.multi INFO: cancelling 16971866 on provider astro 2024-03-10 01:40:53 proc_lsst.multi:146 [INFO] found job 16971862 in provider astro proc_lsst.multi INFO: found job 16971862 in provider astro 2024-03-10 01:40:53 proc_lsst.multi:201 [INFO] cancelling 16971862 on provider astro proc_lsst.multi INFO: cancelling 16971862 on provider astro 2024-03-10 01:40:53 proc_lsst.multi:146 [INFO] found job 16971853 in provider astro proc_lsst.multi INFO: found job 16971853 in provider astro 2024-03-10 01:40:53 proc_lsst.multi:201 [INFO] cancelling 16971853 on provider astro proc_lsst.multi INFO: cancelling 16971853 on provider astro 2024-03-10 01:40:54 proc_lsst.multi:146 [INFO] found job 16971851 in provider astro proc_lsst.multi INFO: found job 16971851 in provider astro 2024-03-10 01:40:54 proc_lsst.multi:201 [INFO] cancelling 16971851 on provider astro proc_lsst.multi INFO: cancelling 16971851 on provider astro 2024-03-10 01:40:54 proc_lsst.multi:146 [INFO] found job 16971848 in provider astro proc_lsst.multi INFO: found job 16971848 in provider astro 2024-03-10 01:40:54 proc_lsst.multi:201 [INFO] cancelling 16971848 on provider astro proc_lsst.multi INFO: cancelling 16971848 on provider astro 2024-03-10 01:40:54 proc_lsst.multi:146 [INFO] found job 16971860 in provider astro proc_lsst.multi INFO: found job 16971860 in provider astro 2024-03-10 01:40:54 proc_lsst.multi:201 [INFO] cancelling 16971860 on provider astro proc_lsst.multi INFO: cancelling 16971860 on provider astro 2024-03-10 01:40:54 proc_lsst.multi:146 [INFO] found job 16971857 in provider astro proc_lsst.multi INFO: found job 16971857 in provider astro 2024-03-10 01:40:54 proc_lsst.multi:201 [INFO] cancelling 16971857 on provider astro proc_lsst.multi INFO: cancelling 16971857 on provider astro 2024-03-10 01:40:54 proc_lsst.multi:146 [INFO] found job 16971867 in provider astro proc_lsst.multi INFO: found job 16971867 in provider astro 2024-03-10 01:40:54 proc_lsst.multi:201 [INFO] cancelling 16971867 on provider astro proc_lsst.multi INFO: cancelling 16971867 on provider astro 2024-03-10 01:40:55 proc_lsst.multi:146 [INFO] found job 16971855 in provider astro proc_lsst.multi INFO: found job 16971855 in provider astro 2024-03-10 01:40:55 proc_lsst.multi:201 [INFO] cancelling 16971855 on provider astro proc_lsst.multi INFO: cancelling 16971855 on provider astro 2024-03-10 01:40:55 proc_lsst.multi:146 [INFO] found job 16971863 in provider astro proc_lsst.multi INFO: found job 16971863 in provider astro 2024-03-10 01:40:55 proc_lsst.multi:201 [INFO] cancelling 16971863 on provider astro proc_lsst.multi INFO: cancelling 16971863 on provider astro 2024-03-10 01:40:55 proc_lsst.multi:146 [INFO] found job 16971865 in provider astro proc_lsst.multi INFO: found job 16971865 in provider astro 2024-03-10 01:40:55 proc_lsst.multi:201 [INFO] cancelling 16971865 on provider astro proc_lsst.multi INFO: cancelling 16971865 on provider astro 2024-03-10 01:40:55 proc_lsst.multi:146 [INFO] found job 16971852 in provider astro proc_lsst.multi INFO: found job 16971852 in provider astro 2024-03-10 01:40:55 proc_lsst.multi:201 [INFO] cancelling 16971852 on provider astro proc_lsst.multi INFO: cancelling 16971852 on provider astro 2024-03-10 01:40:55 proc_lsst.multi:146 [INFO] found job 16971868 in provider astro proc_lsst.multi INFO: found job 16971868 in provider astro 2024-03-10 01:40:55 proc_lsst.multi:201 [INFO] cancelling 16971868 on provider astro proc_lsst.multi INFO: cancelling 16971868 on provider astro 2024-03-10 01:40:55 proc_lsst.multi:146 [INFO] found job 16971870 in provider astro proc_lsst.multi INFO: found job 16971870 in provider astro 2024-03-10 01:40:55 proc_lsst.multi:201 [INFO] cancelling 16971870 on provider astro proc_lsst.multi INFO: cancelling 16971870 on provider astro 2024-03-10 01:40:56 proc_lsst.multi:146 [INFO] found job 16971869 in provider astro proc_lsst.multi INFO: found job 16971869 in provider astro 2024-03-10 01:40:56 proc_lsst.multi:201 [INFO] cancelling 16971869 on provider astro proc_lsst.multi INFO: cancelling 16971869 on provider astro 2024-03-10 01:40:56 proc_lsst.multi:146 [INFO] found job 16971871 in provider astro proc_lsst.multi INFO: found job 16971871 in provider astro 2024-03-10 01:40:56 proc_lsst.multi:201 [INFO] cancelling 16971871 on provider astro proc_lsst.multi INFO: cancelling 16971871 on provider astro 2024-03-10 01:40:56 proc_lsst.multi:146 [INFO] found job 16971843 in provider astro proc_lsst.multi INFO: found job 16971843 in provider astro 2024-03-10 01:40:56 proc_lsst.multi:201 [INFO] cancelling 16971843 on provider astro proc_lsst.multi INFO: cancelling 16971843 on provider astro 2024-03-10 01:40:56 proc_lsst.multi:146 [INFO] found job 16971844 in provider astro proc_lsst.multi INFO: found job 16971844 in provider astro 2024-03-10 01:40:56 proc_lsst.multi:201 [INFO] cancelling 16971844 on provider astro proc_lsst.multi INFO: cancelling 16971844 on provider astro 2024-03-10 01:40:56 proc_lsst.multi:146 [INFO] found job 44401 in provider local proc_lsst.multi INFO: found job 44401 in provider local 2024-03-10 01:40:56 proc_lsst.multi:201 [INFO] cancelling 44401 on provider local proc_lsst.multi INFO: cancelling 44401 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 44401 2024-03-10 01:40:56 proc_lsst.multi:146 [INFO] found job 16971847 in provider astro proc_lsst.multi INFO: found job 16971847 in provider astro 2024-03-10 01:40:56 proc_lsst.multi:201 [INFO] cancelling 16971847 on provider astro proc_lsst.multi INFO: cancelling 16971847 on provider astro 2024-03-10 01:40:57 proc_lsst.multi:146 [INFO] found job 16971874 in provider escience_rtx6k proc_lsst.multi INFO: found job 16971874 in provider escience_rtx6k 2024-03-10 01:40:57 proc_lsst.multi:201 [INFO] cancelling 16971874 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16971874 on provider escience_rtx6k 2024-03-10 01:40:57 proc_lsst.multi:146 [INFO] found job 16971873 in provider escience_rtx6k proc_lsst.multi INFO: found job 16971873 in provider escience_rtx6k 2024-03-10 01:40:57 proc_lsst.multi:201 [INFO] cancelling 16971873 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16971873 on provider escience_rtx6k 2024-03-10 01:40:57 proc_lsst.multi:146 [INFO] found job 16971878 in provider escience_a40 proc_lsst.multi INFO: found job 16971878 in provider escience_a40 2024-03-10 01:40:57 proc_lsst.multi:201 [INFO] cancelling 16971878 on provider escience_a40 proc_lsst.multi INFO: cancelling 16971878 on provider escience_a40 2024-03-10 01:40:57 proc_lsst.multi:146 [INFO] found job 16971881 in provider escience_a40 proc_lsst.multi INFO: found job 16971881 in provider escience_a40 2024-03-10 01:40:57 proc_lsst.multi:201 [INFO] cancelling 16971881 on provider escience_a40 proc_lsst.multi INFO: cancelling 16971881 on provider escience_a40 2024-03-10 01:40:57 proc_lsst.multi:146 [INFO] found job 16971888 in provider escience_a40 proc_lsst.multi INFO: found job 16971888 in provider escience_a40 2024-03-10 01:40:57 proc_lsst.multi:201 [INFO] cancelling 16971888 on provider escience_a40 proc_lsst.multi INFO: cancelling 16971888 on provider escience_a40 2024-03-10 01:40:57 proc_lsst.multi:146 [INFO] found job 16971845 in provider astro proc_lsst.multi INFO: found job 16971845 in provider astro 2024-03-10 01:40:57 proc_lsst.multi:201 [INFO] cancelling 16971845 on provider astro proc_lsst.multi INFO: cancelling 16971845 on provider astro 2024-03-10 01:40:58 proc_lsst.multi:146 [INFO] found job 16971846 in provider astro proc_lsst.multi INFO: found job 16971846 in provider astro 2024-03-10 01:40:58 proc_lsst.multi:201 [INFO] cancelling 16971846 on provider astro proc_lsst.multi INFO: cancelling 16971846 on provider astro 2024-03-10 01:40:58 proc_lsst.multi:146 [INFO] found job 16971880 in provider escience_a40 proc_lsst.multi INFO: found job 16971880 in provider escience_a40 2024-03-10 01:40:58 proc_lsst.multi:201 [INFO] cancelling 16971880 on provider escience_a40 proc_lsst.multi INFO: cancelling 16971880 on provider escience_a40 2024-03-10 01:40:58 proc_lsst.multi:146 [INFO] found job 16971882 in provider escience_a40 proc_lsst.multi INFO: found job 16971882 in provider escience_a40 2024-03-10 01:40:58 proc_lsst.multi:201 [INFO] cancelling 16971882 on provider escience_a40 proc_lsst.multi INFO: cancelling 16971882 on provider escience_a40 2024-03-10 01:40:58 proc_lsst.multi:146 [INFO] found job 16971879 in provider escience_a40 proc_lsst.multi INFO: found job 16971879 in provider escience_a40 2024-03-10 01:40:58 proc_lsst.multi:201 [INFO] cancelling 16971879 on provider escience_a40 proc_lsst.multi INFO: cancelling 16971879 on provider escience_a40 2024-03-10 01:40:58 proc_lsst.multi:146 [INFO] found job 16971887 in provider escience_a40 proc_lsst.multi INFO: found job 16971887 in provider escience_a40 2024-03-10 01:40:58 proc_lsst.multi:201 [INFO] cancelling 16971887 on provider escience_a40 proc_lsst.multi INFO: cancelling 16971887 on provider escience_a40 2024-03-10 01:40:58 proc_lsst.multi:146 [INFO] found job 16971886 in provider escience_a40 proc_lsst.multi INFO: found job 16971886 in provider escience_a40 2024-03-10 01:40:58 proc_lsst.multi:201 [INFO] cancelling 16971886 on provider escience_a40 proc_lsst.multi INFO: cancelling 16971886 on provider escience_a40 2024-03-10 01:40:59 proc_lsst.multi:146 [INFO] found job 16971885 in provider escience_a40 proc_lsst.multi INFO: found job 16971885 in provider escience_a40 2024-03-10 01:40:59 proc_lsst.multi:201 [INFO] cancelling 16971885 on provider escience_a40 proc_lsst.multi INFO: cancelling 16971885 on provider escience_a40 2024-03-10 01:40:59 proc_lsst.multi:146 [INFO] found job 16971884 in provider escience_a40 proc_lsst.multi INFO: found job 16971884 in provider escience_a40 2024-03-10 01:40:59 proc_lsst.multi:201 [INFO] cancelling 16971884 on provider escience_a40 proc_lsst.multi INFO: cancelling 16971884 on provider escience_a40 2024-03-10 01:40:59 proc_lsst.multi:146 [INFO] found job 16971883 in provider escience_a40 proc_lsst.multi INFO: found job 16971883 in provider escience_a40 2024-03-10 01:40:59 proc_lsst.multi:201 [INFO] cancelling 16971883 on provider escience_a40 proc_lsst.multi INFO: cancelling 16971883 on provider escience_a40 2024-03-10 01:40:59 proc_lsst.multi:146 [INFO] found job 16971877 in provider escience_rtx6k proc_lsst.multi INFO: found job 16971877 in provider escience_rtx6k 2024-03-10 01:40:59 proc_lsst.multi:201 [INFO] cancelling 16971877 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16971877 on provider escience_rtx6k 2024-03-10 01:40:59 proc_lsst.multi:146 [INFO] found job 16971875 in provider escience_rtx6k proc_lsst.multi INFO: found job 16971875 in provider escience_rtx6k 2024-03-10 01:40:59 proc_lsst.multi:201 [INFO] cancelling 16971875 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16971875 on provider escience_rtx6k 2024-03-10 01:40:59 proc_lsst.multi:146 [INFO] found job 16971872 in provider escience_rtx6k proc_lsst.multi INFO: found job 16971872 in provider escience_rtx6k 2024-03-10 01:40:59 proc_lsst.multi:201 [INFO] cancelling 16971872 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16971872 on provider escience_rtx6k 2024-03-10 01:41:00 proc_lsst.multi:146 [INFO] found job 16971842 in provider astro proc_lsst.multi INFO: found job 16971842 in provider astro 2024-03-10 01:41:00 proc_lsst.multi:201 [INFO] cancelling 16971842 on provider astro proc_lsst.multi INFO: cancelling 16971842 on provider astro parsl.dataflow.dflow INFO: Shutting down executor multi 2024-03-10 01:41:00 proc_lsst.multi:40 [INFO] Cancelling all provider resources proc_lsst.multi INFO: Cancelling all provider resources 2024-03-10 01:41:00 proc_lsst.multi:47 [INFO] new jobs since last cancel ['16971842', '16971843', '16971844', '16971845', '16971846', '16971847', '16971848', '16971849', '16971850', '16971851', '16971852', '16971853', '16971854', '16971855', '16971856', '16971857', '16971858', '16971859', '16971860', '16971861', '16971862', '16971863', '16971864', '16971865', '16971866', '16971867', '16971868', '16971869', '16971870', '16971871', '16971878', '16971879', '16971880', '16971881', '16971882', '16971883', '16971884', '16971885', '16971886', '16971887', '16971888', '16971872', '16971873', '16971874', '16971875', '16971877', '44401'] proc_lsst.multi INFO: new jobs since last cancel ['16971842', '16971843', '16971844', '16971845', '16971846', '16971847', '16971848', '16971849', '16971850', '16971851', '16971852', '16971853', '16971854', '16971855', '16971856', '16971857', '16971858', '16971859', '16971860', '16971861', '16971862', '16971863', '16971864', '16971865', '16971866', '16971867', '16971868', '16971869', '16971870', '16971871', '16971878', '16971879', '16971880', '16971881', '16971882', '16971883', '16971884', '16971885', '16971886', '16971887', '16971888', '16971872', '16971873', '16971874', '16971875', '16971877', '44401'] 2024-03-10 01:41:00 proc_lsst.multi:146 [INFO] found job 16971842 in provider astro proc_lsst.multi INFO: found job 16971842 in provider astro 2024-03-10 01:41:00 proc_lsst.multi:201 [INFO] cancelling 16971842 on provider astro proc_lsst.multi INFO: cancelling 16971842 on provider astro 2024-03-10 01:41:00 proc_lsst.multi:146 [INFO] found job 16971843 in provider astro proc_lsst.multi INFO: found job 16971843 in provider astro 2024-03-10 01:41:00 proc_lsst.multi:201 [INFO] cancelling 16971843 on provider astro proc_lsst.multi INFO: cancelling 16971843 on provider astro 2024-03-10 01:41:00 proc_lsst.multi:146 [INFO] found job 16971844 in provider astro proc_lsst.multi INFO: found job 16971844 in provider astro 2024-03-10 01:41:00 proc_lsst.multi:201 [INFO] cancelling 16971844 on provider astro proc_lsst.multi INFO: cancelling 16971844 on provider astro 2024-03-10 01:41:00 proc_lsst.multi:146 [INFO] found job 16971845 in provider astro proc_lsst.multi INFO: found job 16971845 in provider astro 2024-03-10 01:41:00 proc_lsst.multi:201 [INFO] cancelling 16971845 on provider astro proc_lsst.multi INFO: cancelling 16971845 on provider astro 2024-03-10 01:41:01 proc_lsst.multi:146 [INFO] found job 16971846 in provider astro proc_lsst.multi INFO: found job 16971846 in provider astro 2024-03-10 01:41:01 proc_lsst.multi:201 [INFO] cancelling 16971846 on provider astro proc_lsst.multi INFO: cancelling 16971846 on provider astro 2024-03-10 01:41:01 proc_lsst.multi:146 [INFO] found job 16971847 in provider astro proc_lsst.multi INFO: found job 16971847 in provider astro 2024-03-10 01:41:01 proc_lsst.multi:201 [INFO] cancelling 16971847 on provider astro proc_lsst.multi INFO: cancelling 16971847 on provider astro 2024-03-10 01:41:01 proc_lsst.multi:146 [INFO] found job 16971848 in provider astro proc_lsst.multi INFO: found job 16971848 in provider astro 2024-03-10 01:41:01 proc_lsst.multi:201 [INFO] cancelling 16971848 on provider astro proc_lsst.multi INFO: cancelling 16971848 on provider astro 2024-03-10 01:41:01 proc_lsst.multi:146 [INFO] found job 16971849 in provider astro proc_lsst.multi INFO: found job 16971849 in provider astro 2024-03-10 01:41:01 proc_lsst.multi:201 [INFO] cancelling 16971849 on provider astro proc_lsst.multi INFO: cancelling 16971849 on provider astro 2024-03-10 01:41:01 proc_lsst.multi:146 [INFO] found job 16971850 in provider astro proc_lsst.multi INFO: found job 16971850 in provider astro 2024-03-10 01:41:01 proc_lsst.multi:201 [INFO] cancelling 16971850 on provider astro proc_lsst.multi INFO: cancelling 16971850 on provider astro 2024-03-10 01:41:01 proc_lsst.multi:146 [INFO] found job 16971851 in provider astro proc_lsst.multi INFO: found job 16971851 in provider astro 2024-03-10 01:41:01 proc_lsst.multi:201 [INFO] cancelling 16971851 on provider astro proc_lsst.multi INFO: cancelling 16971851 on provider astro 2024-03-10 01:41:02 proc_lsst.multi:146 [INFO] found job 16971852 in provider astro proc_lsst.multi INFO: found job 16971852 in provider astro 2024-03-10 01:41:02 proc_lsst.multi:201 [INFO] cancelling 16971852 on provider astro proc_lsst.multi INFO: cancelling 16971852 on provider astro 2024-03-10 01:41:02 proc_lsst.multi:146 [INFO] found job 16971853 in provider astro proc_lsst.multi INFO: found job 16971853 in provider astro 2024-03-10 01:41:02 proc_lsst.multi:201 [INFO] cancelling 16971853 on provider astro proc_lsst.multi INFO: cancelling 16971853 on provider astro 2024-03-10 01:41:02 proc_lsst.multi:146 [INFO] found job 16971854 in provider astro proc_lsst.multi INFO: found job 16971854 in provider astro 2024-03-10 01:41:02 proc_lsst.multi:201 [INFO] cancelling 16971854 on provider astro proc_lsst.multi INFO: cancelling 16971854 on provider astro 2024-03-10 01:41:02 proc_lsst.multi:146 [INFO] found job 16971855 in provider astro proc_lsst.multi INFO: found job 16971855 in provider astro 2024-03-10 01:41:02 proc_lsst.multi:201 [INFO] cancelling 16971855 on provider astro proc_lsst.multi INFO: cancelling 16971855 on provider astro 2024-03-10 01:41:02 proc_lsst.multi:146 [INFO] found job 16971856 in provider astro proc_lsst.multi INFO: found job 16971856 in provider astro 2024-03-10 01:41:02 proc_lsst.multi:201 [INFO] cancelling 16971856 on provider astro proc_lsst.multi INFO: cancelling 16971856 on provider astro 2024-03-10 01:41:03 proc_lsst.multi:146 [INFO] found job 16971857 in provider astro proc_lsst.multi INFO: found job 16971857 in provider astro 2024-03-10 01:41:03 proc_lsst.multi:201 [INFO] cancelling 16971857 on provider astro proc_lsst.multi INFO: cancelling 16971857 on provider astro 2024-03-10 01:41:03 proc_lsst.multi:146 [INFO] found job 16971858 in provider astro proc_lsst.multi INFO: found job 16971858 in provider astro 2024-03-10 01:41:03 proc_lsst.multi:201 [INFO] cancelling 16971858 on provider astro proc_lsst.multi INFO: cancelling 16971858 on provider astro 2024-03-10 01:41:03 proc_lsst.multi:146 [INFO] found job 16971859 in provider astro proc_lsst.multi INFO: found job 16971859 in provider astro 2024-03-10 01:41:03 proc_lsst.multi:201 [INFO] cancelling 16971859 on provider astro proc_lsst.multi INFO: cancelling 16971859 on provider astro 2024-03-10 01:41:03 proc_lsst.multi:146 [INFO] found job 16971860 in provider astro proc_lsst.multi INFO: found job 16971860 in provider astro 2024-03-10 01:41:03 proc_lsst.multi:201 [INFO] cancelling 16971860 on provider astro proc_lsst.multi INFO: cancelling 16971860 on provider astro 2024-03-10 01:41:03 proc_lsst.multi:146 [INFO] found job 16971861 in provider astro proc_lsst.multi INFO: found job 16971861 in provider astro 2024-03-10 01:41:03 proc_lsst.multi:201 [INFO] cancelling 16971861 on provider astro proc_lsst.multi INFO: cancelling 16971861 on provider astro 2024-03-10 01:41:03 proc_lsst.multi:146 [INFO] found job 16971862 in provider astro proc_lsst.multi INFO: found job 16971862 in provider astro 2024-03-10 01:41:03 proc_lsst.multi:201 [INFO] cancelling 16971862 on provider astro proc_lsst.multi INFO: cancelling 16971862 on provider astro 2024-03-10 01:41:04 proc_lsst.multi:146 [INFO] found job 16971863 in provider astro proc_lsst.multi INFO: found job 16971863 in provider astro 2024-03-10 01:41:04 proc_lsst.multi:201 [INFO] cancelling 16971863 on provider astro proc_lsst.multi INFO: cancelling 16971863 on provider astro 2024-03-10 01:41:04 proc_lsst.multi:146 [INFO] found job 16971864 in provider astro proc_lsst.multi INFO: found job 16971864 in provider astro 2024-03-10 01:41:04 proc_lsst.multi:201 [INFO] cancelling 16971864 on provider astro proc_lsst.multi INFO: cancelling 16971864 on provider astro 2024-03-10 01:41:04 proc_lsst.multi:146 [INFO] found job 16971865 in provider astro proc_lsst.multi INFO: found job 16971865 in provider astro 2024-03-10 01:41:04 proc_lsst.multi:201 [INFO] cancelling 16971865 on provider astro proc_lsst.multi INFO: cancelling 16971865 on provider astro 2024-03-10 01:41:04 proc_lsst.multi:146 [INFO] found job 16971866 in provider astro proc_lsst.multi INFO: found job 16971866 in provider astro 2024-03-10 01:41:04 proc_lsst.multi:201 [INFO] cancelling 16971866 on provider astro proc_lsst.multi INFO: cancelling 16971866 on provider astro 2024-03-10 01:41:04 proc_lsst.multi:146 [INFO] found job 16971867 in provider astro proc_lsst.multi INFO: found job 16971867 in provider astro 2024-03-10 01:41:04 proc_lsst.multi:201 [INFO] cancelling 16971867 on provider astro proc_lsst.multi INFO: cancelling 16971867 on provider astro 2024-03-10 01:41:04 proc_lsst.multi:146 [INFO] found job 16971868 in provider astro proc_lsst.multi INFO: found job 16971868 in provider astro 2024-03-10 01:41:04 proc_lsst.multi:201 [INFO] cancelling 16971868 on provider astro proc_lsst.multi INFO: cancelling 16971868 on provider astro 2024-03-10 01:41:05 proc_lsst.multi:146 [INFO] found job 16971869 in provider astro proc_lsst.multi INFO: found job 16971869 in provider astro 2024-03-10 01:41:05 proc_lsst.multi:201 [INFO] cancelling 16971869 on provider astro proc_lsst.multi INFO: cancelling 16971869 on provider astro 2024-03-10 01:41:05 proc_lsst.multi:146 [INFO] found job 16971870 in provider astro proc_lsst.multi INFO: found job 16971870 in provider astro 2024-03-10 01:41:05 proc_lsst.multi:201 [INFO] cancelling 16971870 on provider astro proc_lsst.multi INFO: cancelling 16971870 on provider astro 2024-03-10 01:41:05 proc_lsst.multi:146 [INFO] found job 16971871 in provider astro proc_lsst.multi INFO: found job 16971871 in provider astro 2024-03-10 01:41:05 proc_lsst.multi:201 [INFO] cancelling 16971871 on provider astro proc_lsst.multi INFO: cancelling 16971871 on provider astro 2024-03-10 01:41:05 proc_lsst.multi:146 [INFO] found job 16971878 in provider escience_a40 proc_lsst.multi INFO: found job 16971878 in provider escience_a40 2024-03-10 01:41:05 proc_lsst.multi:201 [INFO] cancelling 16971878 on provider escience_a40 proc_lsst.multi INFO: cancelling 16971878 on provider escience_a40 2024-03-10 01:41:05 proc_lsst.multi:146 [INFO] found job 16971879 in provider escience_a40 proc_lsst.multi INFO: found job 16971879 in provider escience_a40 2024-03-10 01:41:05 proc_lsst.multi:201 [INFO] cancelling 16971879 on provider escience_a40 proc_lsst.multi INFO: cancelling 16971879 on provider escience_a40 2024-03-10 01:41:05 proc_lsst.multi:146 [INFO] found job 16971880 in provider escience_a40 proc_lsst.multi INFO: found job 16971880 in provider escience_a40 2024-03-10 01:41:05 proc_lsst.multi:201 [INFO] cancelling 16971880 on provider escience_a40 proc_lsst.multi INFO: cancelling 16971880 on provider escience_a40 2024-03-10 01:41:07 proc_lsst.multi:146 [INFO] found job 16971881 in provider escience_a40 proc_lsst.multi INFO: found job 16971881 in provider escience_a40 2024-03-10 01:41:07 proc_lsst.multi:201 [INFO] cancelling 16971881 on provider escience_a40 proc_lsst.multi INFO: cancelling 16971881 on provider escience_a40 2024-03-10 01:41:08 proc_lsst.multi:146 [INFO] found job 16971882 in provider escience_a40 proc_lsst.multi INFO: found job 16971882 in provider escience_a40 2024-03-10 01:41:08 proc_lsst.multi:201 [INFO] cancelling 16971882 on provider escience_a40 proc_lsst.multi INFO: cancelling 16971882 on provider escience_a40 2024-03-10 01:41:08 proc_lsst.multi:146 [INFO] found job 16971883 in provider escience_a40 proc_lsst.multi INFO: found job 16971883 in provider escience_a40 2024-03-10 01:41:08 proc_lsst.multi:201 [INFO] cancelling 16971883 on provider escience_a40 proc_lsst.multi INFO: cancelling 16971883 on provider escience_a40 2024-03-10 01:41:08 proc_lsst.multi:146 [INFO] found job 16971884 in provider escience_a40 proc_lsst.multi INFO: found job 16971884 in provider escience_a40 2024-03-10 01:41:08 proc_lsst.multi:201 [INFO] cancelling 16971884 on provider escience_a40 proc_lsst.multi INFO: cancelling 16971884 on provider escience_a40 2024-03-10 01:41:08 proc_lsst.multi:146 [INFO] found job 16971885 in provider escience_a40 proc_lsst.multi INFO: found job 16971885 in provider escience_a40 2024-03-10 01:41:08 proc_lsst.multi:201 [INFO] cancelling 16971885 on provider escience_a40 proc_lsst.multi INFO: cancelling 16971885 on provider escience_a40 2024-03-10 01:41:08 proc_lsst.multi:146 [INFO] found job 16971886 in provider escience_a40 proc_lsst.multi INFO: found job 16971886 in provider escience_a40 2024-03-10 01:41:08 proc_lsst.multi:201 [INFO] cancelling 16971886 on provider escience_a40 proc_lsst.multi INFO: cancelling 16971886 on provider escience_a40 2024-03-10 01:41:08 proc_lsst.multi:146 [INFO] found job 16971887 in provider escience_a40 proc_lsst.multi INFO: found job 16971887 in provider escience_a40 2024-03-10 01:41:08 proc_lsst.multi:201 [INFO] cancelling 16971887 on provider escience_a40 proc_lsst.multi INFO: cancelling 16971887 on provider escience_a40 2024-03-10 01:41:09 proc_lsst.multi:146 [INFO] found job 16971888 in provider escience_a40 proc_lsst.multi INFO: found job 16971888 in provider escience_a40 2024-03-10 01:41:09 proc_lsst.multi:201 [INFO] cancelling 16971888 on provider escience_a40 proc_lsst.multi INFO: cancelling 16971888 on provider escience_a40 2024-03-10 01:41:09 proc_lsst.multi:146 [INFO] found job 16971872 in provider escience_rtx6k proc_lsst.multi INFO: found job 16971872 in provider escience_rtx6k 2024-03-10 01:41:09 proc_lsst.multi:201 [INFO] cancelling 16971872 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16971872 on provider escience_rtx6k 2024-03-10 01:41:09 proc_lsst.multi:146 [INFO] found job 16971873 in provider escience_rtx6k proc_lsst.multi INFO: found job 16971873 in provider escience_rtx6k 2024-03-10 01:41:09 proc_lsst.multi:201 [INFO] cancelling 16971873 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16971873 on provider escience_rtx6k 2024-03-10 01:41:09 proc_lsst.multi:146 [INFO] found job 16971874 in provider escience_rtx6k proc_lsst.multi INFO: found job 16971874 in provider escience_rtx6k 2024-03-10 01:41:09 proc_lsst.multi:201 [INFO] cancelling 16971874 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16971874 on provider escience_rtx6k 2024-03-10 01:41:09 proc_lsst.multi:146 [INFO] found job 16971875 in provider escience_rtx6k proc_lsst.multi INFO: found job 16971875 in provider escience_rtx6k 2024-03-10 01:41:09 proc_lsst.multi:201 [INFO] cancelling 16971875 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16971875 on provider escience_rtx6k 2024-03-10 01:41:09 proc_lsst.multi:146 [INFO] found job 16971877 in provider escience_rtx6k proc_lsst.multi INFO: found job 16971877 in provider escience_rtx6k 2024-03-10 01:41:09 proc_lsst.multi:201 [INFO] cancelling 16971877 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16971877 on provider escience_rtx6k 2024-03-10 01:41:10 proc_lsst.multi:146 [INFO] found job 44401 in provider local proc_lsst.multi INFO: found job 44401 in provider local 2024-03-10 01:41:10 proc_lsst.multi:201 [INFO] cancelling 44401 on provider local proc_lsst.multi INFO: cancelling 44401 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 44401 parsl.providers.local.local WARNING: Failed to kill PID: 44401 and child processes on local 2024-03-10 01:41:11 proc_lsst.multi:50 [INFO] no new jobs since last cancel, resuming executor shutdown proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt parsl.dataflow.dflow INFO: Shut down executor multi parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete parsl.executors.threads DEBUG: Done with executor shutdown parsl.dataflow.dflow INFO: Shut down executor _parsl_internal parsl.dataflow.dflow INFO: Terminated executors parsl.dataflow.dflow INFO: DFK cleanup complete parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 221.5463 seconds lsst.ctrl.bps.drivers INFO: Run 'DEEP_20190601_A1c_science#step0_20240310T013701Z' submitted for execution with id 'None' lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 221.5582 seconds; current memory usage: 0.272 Gibyte, delta: 0.000 Gibyte, peak delta: 0.002 Gibyte lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 243.9258 seconds; current memory usage: 0.272 Gibyte, delta: 0.135 Gibyte, peak delta: 0.137 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.274 Gibyte (main), 0.274 Gibyte (largest child process) Run Id: None Run Name: DEEP_20190601_A1c_science#step0_20240310T013701Z parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up real 4m6.688s user 0m54.619s sys 0m25.646s